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Foreword 

The  profound  revolution  in  healthcare,  from  managed  care  to  gene-based 
therapies  to  artificial  organs  to  robotic  surgery  to  virtual  reality  to  microelectronic 
mechanical  systems  (MEMS)  and  nanotechnology,  has  disrupted  the  entire  practice 
of  medicine  and  surgery  in  an  incredibly  short  period  of  time.  It  is  no  wonder  that 
the  public,  which  is  becoming  more  educated  through  the  media  and  the  Internet, 
has  come  to  question  the  competency  of  the  practioners.  In  November,  1999,  the 
Institute  of  Medicine  completed  its  report  “To  Err  Is  Human:  Building  a  Safer 
Health  System”.  In  this  report,  it  was  estimated  that  from  44,000  to  98,000  patients 
died  due  to  medical  errors.  However,  it  was  not  so  much  single  errors  by  an 
individual,  but  rather  inherent  weaknesses  in  the  complex  healthcare  system  itself 
that  led  inevitably  to  competent  and  well-intentioned  healthcare  providers’  actions 
that  resulted  in  errors.  One  of  the  significant  shortfalls  identified  was  that  of 
education  and  training.  It  was  also  recommended  that  new  technologies,  such  as 
simulation  and  objective  assessment,  are  powerful  instruments  that  could  improve 
physician  competence  and  the  quality  of  patient  care. 

For  over  50  years  the  aviation,  aerospace,  maritime,  military,  nuclear  energy 
and  other  high  risk  professions  have  been  using  simulators  for  training  difficult  and 
demanding  tasks.  In  so  doing,  these  industries  have  reduced  errors  to  nearly  zero. 
Since  1955,  examination  on  a  flight  simulator  has  been  required  by  the  aviation 
industry  and  military  as  a  component  for  certification  and  annual  re-certification. 
Although  the  simulation  and  objective  assessment  for  medical  and  surgical 
procedures  is  in  its  infancy,  there  are  some  technologies  and  methodologies  that 
have  been  proven  to  be  effective. 

However,  there  are  a  number  of  problems  which  must  be  addressed  during 
this  initial  period  of  objective  assessment  with  surgical  simulators.  First  and 
foremost,  there  is  no  agreed  upon  set  of  definitions  of  terms,  no  classification  of  the 
tasks  that  are  being  used  for  training  and  evaluation,  and  no  curriculum  to 
accompany  the  training.  Although  numerous  investigators  have  excellent  systems 
for  training  and  evaluation,  there  is  no  commonality  that  permits  comparison  of 
results.  Finally,  there  are  no  studies  relating  the  training  and  assessment  methods 
with  patient  outcomes. 

This  workshop  specifically  addresses  the  above  mention  critical 
infrastructure  issues  that  will  permit  the  science  of  surgical  education  to  move  into 
an  era  of  objective  assessment  of  surgical  skills.  It  is  clear  that  this  is  just  a  first 
step,  but  the  technology  is  maturing  rapidly  enough  that  the  medical  profession 
should  be  able  to  accomplish  the  same  quality  of  safety  as  other  industries  in  a 
much  shorter  timeframe.  This  workshop  also  acknowledges  that  surgical  technical 
skills  are  but  a  portion  of  the  overall  development  and  assessment  of  competency; 
however  by  providing  a  stringently  validated  component  of  technical  skills,  the 
overall  quality  of  competency  should  be  raised  to  a  level  that  significantly  reduces 
surgical  errors  and  thereby  improves  patient  safety. 
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Metrics  for  Objective  Assessment  of  Surgical  Skills  Workshop 

Scottsdale,  AZ 
9-10  July,  2001 


Executive  Summary 

On  9-10  July,  2001  the  Metrics  for  Objective  Assessment  of  Surgical  Skills 
Workshop  convened  an  international  assemblage  of  subject  matter  experts  in 
objective  assessment  of  surgical  technical  skills  and  of  representatives  of  relevant 
official  bodies  involved  in  surgical  education,  evaluation  and  certification.  There 
have  been  recent  advances  in  the  educational  science  of  objective  assessment  and 
the  technology  of  medical  modeling  and  simulation  (with  surrogate  tissue,  abstract 
objects  or  virtual  reality  (VR)  systems)  for  training  of  technical  skills.  In  addition, 
focus  on  identification  and  prevention  of  medical  errors,  the  need  for  objective 
criteria  for  assessment  of  surgical  skills  and  the  increasing  demand  for 
accountability  to  the  public  has  revealed  that  there  is  no  infrastructure  for  the 
objective  assessment  of  technical  skills.  Several  investigators  are  validating  many 
different  systems  for  training  and  evaluation,  using  different  tests,  criteria, 
validation  methods  and  even  nomenclature.  This  workshop  is  an  attempt  to 
establish  a  standardization  of  nomenclature  and  assessment  methodologies  so  the 
surgical  education,  training  and  evaluation  community  can  communicate  with  a 
common  language  and  have  a  common  basis  for  comparing  statistics  from 
different  centers.  The  results  of  this  workshop  are  to  be  considered  a  first  order 
approximation  from  the  community  of  subject  matter  experts  that  can  provide  a 
“straw  man”  for  future  refinement. 

Purpose:  The  purpose  was  to  establish  a  consensus  on  a  baseline  set  of  metrics  from  which 
future  education,  training,  evaluation  and  research  in  the  technical  aspect  of  surgical  and  procedural 
skills  can  be  measured.  These  metrics  are  anticipated  to  be  useful  in  the  full  spectrum  of  technical 
skills,  from  initial  evaluation  of  applicants,  to  objective  feedback  during  training,  to  on-going 
evaluation  of  performance,  to  criterion  for  certification  and  as  such  can  provide  an  educational  “tool 
kit”  which  surgical  educators  can  utilize  for  their  training  programs. 

Goals :  The  goals  were:  To  identify  the  validated,  relevant  measurements  for  technical 
surgical  skills,  to  evaluate  the  currently  available  systems  that  have  been  validated  in  peer-reviewed 
publications,  and  to  propose  a  set  of  objective  training  and  evaluation  tools  for  technical  surgical 
skills  that  can  be  utilized  by  surgical  educators  and  surgical  training  program  directors  (or  “program 
directors”). 
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Objectives:  The  specific  objectives  were  to  derive:  Definitions,  taxonomy,  analysis  of 
current  systems,  functional  components  for  a  core  curriculum,  list  of  validated  systems  available  for  a 
core  curriculum  and  identification  of  areas  that  require  further  research. 

Scope:  It  should  be  stressed  at  the  outset  that  the  objective  assessment  of  technical  skills 
constitutes  only  one  of  the  range  of  competencies  needed  by  a  medical/  surgical  professionals,  and 
cannot,  by  itself,  be  used  for  certification  of  ability  to  practice  medicine/  surgery  at  any  independent/ 
unsupervised  level.  Technical  operative  competence,  though  of  crucial  importance  in  surgery,  is  not 
enough  and  must  be  accompanied  by  other  skills,  e.g.,  cognitive,  clinical  and  humanistic  etc.,  to 
result  in  a  complete  competent,  safe,  effective  and  caring  professional.  It  is  fully  acknowledged  that 
the  measurement  tools  herein  derived  are  limited  to  the  technical  skills,  and  do  not,  cover  these  other 
equally  important  aspects  of  professional  competence. 

Methodology;.  The  derivation  of  the  data  followed  a  modified  Delphi  methodology  using  the 
experts  indicated  above  (Appendix  1).  Consensus  on  definitions  (Appendix  2,3)  led  to  the 
establishment  of  a  taxonomy  of  abilities,  skills,  tasks  and  procedures  (Appendix  4)  that  comprise  the 
lexicon  of  technical  skills  assessment;  this  is  a  first  approximation  which  will  require  further  vetting 
to  become  all  inclusive.  The  criteria  of  validity  (face,  concurrent,  construct,  content,  and  predictive) 
and  reliability  (inter-rater  and  test-retest)  were  reviewed  for  the  candidate  systems  that  have  been 
reported  in  the  literature.  Subsequent  steps  derived  matrices  that  identified  which  of  the  currently 
validated  objective  assessment  systems  (Appendix  5)  corresponded  to  the  various  abilities,  skills, 
tasks  and  procedures,  and  which  systems  could  be  used  for  different  levels  of  training  and  evaluation 
(basic,  intermediate  and  advanced)  (Appendix  6).  Also  presented  was  a  definition  of  levels  of  overall 
competence  (Appendix  7)  which  could  be  applicable  to  the  demonstration  of  technical  skills.  This 
five-level  hierarchy  is:  Novice,  competent,  proficient,  expert  and  master. 

Results.  The  results  fall  into  five  domains:  Definitions,  taxonomy,  list  of  systems,  levels  for 
curriculum  development  and  research  opportunities  (Appendix  8). 

(a.)  Definitions.  Within  the  definitions  process  there  was  extensive  discussion  regarding  the 
appropriateness  of  the  term  “abilities”.  It  was  recognized  that  there  are  numerous  social,  political  and 
legal  implications  attached  to  the  connotation  of  the  term.  However  a  large  body  of  scientific 
literature  with  specific  relevance  for  the  assessment  process  decided  the  use  of  the  term  ability.  The 
following  are  some  of  the  essential  definitions  for  objective  assessment  (Appendix  2): 
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Ability  The  natural  state  or  condition  of  being  capable;  innate  aptitude  (prior  to 
training)  which  an  individual  brings  to  a  given  task 
Skill  Acquired  (by  training)  proficiency  and  execution  in  some  art,  craft,  or  the  like 

Task  A  piece  of  work  to  be  done;  a  difficult  or  tedious  undertaking  usually 

incorporating  multiple  skills 
Procedure  A  series  of  steps  taken  to  accomplish  an  end 

(b.)  Taxonomy:  Once  the  definitions  were  agreed  upon,  a  classification  (taxonomy)  of  the 
various  types  of  abilities,  skills,  tasks  and  procedures  was  proposed  based  upon  the  measurements  f 
objective  assessment  that  have  been  reported  in  the  medical  literature  on  objective  assessment.  It  is 
acknowledged  that  this  is  not  a  comprehensive  list,  since  it  does  not  include  known  measurements 
from  non-medical  literature  that  have  not  been  validated  by  medical  education  researchers  nor  does  it 
address  the  numerous  procedures  from  the  vast  array  of  specialties  and  sub-specialties  which  have 
specific  operations  and  procedures.  In  addition,  the  state-of-the-art  in  simulation  technology  permits 
the  high  fidelity  simulation  (whether  real  or  virtual  simulations)  of  only  a  few  complete  procedures. 
Nevertheless,  the  workshop  has  provided  a  rather  comprehensive  set  of  assessment  tools  for  basic  and 
intermediate  surgical  skills  training. 

(c.)  Analysis  of  Current  Systems.  Initially,  the  Department  of  Defense  (DoD)  simulation 
program,  which  has  a  major  medical  modeling  and  simulation  (MM&S)  initiative  by  the  US  Army 
Medical  Research  and  Material  Command’s  Telemedicine  and  Advanced  Technology  Research 
Center  (TATRC),  was  reviewed  to  understand  how  a  single,  integrated,  large-scale  effort  in  surgical 
simulation  is  managed.  Subsequently,  following  a  presentation  of  the  validated  systems  for  surgical 
skill  education  and  training  in  the  United  States,  Canada,  Europe  and  Australasia,  the  component 
exercises  of  these  systems  were  analyzed  to  see  which  exercises  measured  fundamental  abilities, 
technical  skills,  complex  tasks  and  full  procedures  (Appendix  5).  It  was  evident  that  many  of  the 
existing  systems  measure  the  same  or  similar  skills,  and  therefore  can  be  expected  to  provide  a  basis 
for  comparison  of  results.  There  were  also  a  few  measurements  considered  important,  such  as  tissue 
handling,  for  which  there  were  no  exercises  or  tests.  This  provides  an  insight  into  areas  for  future 
research  (Appendix  8).  The  vast  majority  of  the  tests  are  for  skills  and  tasks,  with  only  a  few  for 
fundamental  abilities  or  for  total  procedures.  The  implications  are  that  the  large  pool  of  scientific 
research  on  abilities  for  safety  in  the  aviation,  transportation,  nuclear  power,  military  and  other  fields 
need  to  be  investigated  for  medical  applications.  Likewise,  it  was  noted  that  there  were  few 
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complete  surgical  procedures  that  have  been  simulated,  and  those  which  do  exist  are  not  of  high 
fidelity.  Nevertheless  it  was  felt  that  the  science  of  simulating  tissue  from  synthetic  materials  or  in 
virtual  reality  will  continue  to  rapidly  advance  and  that  within  a  decade  or  two  it  will  be  possible  to 
have  simulations  of  simple  procedures  that  are  extremely  close  to  actual  procedures.  There  are  some 
early  successes  in  anesthesiology  and  ultrasound  (with  a  mannequin),  endoscopy  (colonoscopy,  EGD, 
ENT  Sinusoscopy)  and  interventional  radiology  (coronary  angioplasty  and  stent  placement)  that  were 
noted.  It  was  repeatedly  emphasized  that  the  advantage  of  the  simulators  (especially  virtual  reality 
ones)  is  that  as  the  simulators  are  being  used  for  training,  they  are  automatically  objectively  assessing 
performance,  providing  real-time  feedback  (proximate  learning),  continuous  tracking  of  the  student’s 
improvement  (the  learning  curve)  and  providing  a  quantative  score  for  record.  They  can  also  be  used 
for  summative  assessment  of  skills  in  surgical  examination. 

(d.)  Curriculum  development.  The  original  objective  of  trying  to  provide  a  model  core 
curriculum  was  a  bit  over  ambitious.  It  is  apparent  that  there  are  numerous  methods  that  can  be 
developed  that  would  result  in  a  valid  core  curriculum.  Much  discussion  centered  around  1)  what  the 
curriculum  would  be  used  for  -  initial  evaluation  of  applicants,  training  of  students,  evaluation  of 
progress  and/or  minimal  performance  standards  to  progress  from  one  level  of  skills  proficiency  to 
another  and  ultimately  as  a  basis  for  determining  competence  by  proper  authorities  in  certification. 

2)  for  whom  the  curriculum  was  developed  -  the  student,  the  program  director,  external  reviewers  or 
certification  bodies  and  3)  who  would  decide  the  correct  performance  criterion  -  program  directors, 
independent  consensus  workshops,  training  (e.g.  Residency  Review  Committee)  or  certification 
bodies.  Ultimately  it  was  agreed  that  the  first  implementation  would  be  structured  for  the  program 
directors  as  an  aid  in  training  and  performance  evaluation  of  the  students.  This  resulted  in  re¬ 
distribution  of  the  skills,  tasks  and  procedures  into  three  levels  -  basic,  intermediate  and  advanced  - 
which  the  program  directors  could  use  and  customize  to  the  needs  of  their  specific  training  programs. 
It  was  noted  that  the  United  Kingdom  (as  represented  by  the  three  countries  of  England,  Ireland  and 
Scotland)  has  examinations  at  three  levels,  which  closely  approximates  the  levels  independently 
defined  by  this  workshop.  One  of  the  limitations  in  trying  to  decide  a  core  set  of  exercises  for  a 
“model  curriculum”  was  that  much  of  the  current  data  has  not  been  analyzed  in  such  a  way  that 
enables  different  systems  to  be  compared  for  reliability  and  validity,  leaving  the  assumption  that  most 
were  equally  valid.  This  certainly  points  to  a  need,  which  is  to  develop  a  system  to  ensure 
comparability  of  results. 
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(d.)  Research  Opportunities.  There  were  three  distinct  areas  for  research  that  were  identified 
during  the  workshop.  1).  A  number  of  surgeons  identified  a  need  for  a  skill  called  “tissue  handling”. 
None  of  the  systems  measure  such  a  skill.  2).  There  are  only  a  few  exercises  which  evaluate 
fundamental  abilities;  it  is  known  that  there  are  many  such  tests  available  in  the  non-medical  arena 
and  therefore  these  should  be  identified  and  validated  for  medical  applications.  3).  Comparison  and 
integration  of  the  various  available  exercises  of  the  different  systems  into  a  single  coherent  “core 
curriculum”.  In  addition,  research  in  haptic  abilities,  as  well  as  the  fundamental  abilities,  are  in  their 
infancy  and  need  more  basic  and  applied  research. 

Future  Directions.  With  the  completion  of  this  workshop  report,  it  is  proposed  to  conduct  an 
Open  Forum  on  Metrics  for  Objective  Assessment  of  Surgical  Skills.  The  purpose  of  this  report  will 
discuss  and  vet  the  original  results  of  this  workshop.  A  suggested  time  frame  is  contained  in 
Appendix  9. 
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Metrics  for  Objective  Assessment  of  Surgical  Skills  Workshop 

Scottsdale,  AZ 
9-10  July,  2001 


On  9-10  July,  2001  the  Metrics  for  Objective  Assessment  of  Surgical  Skills  Workshop 
convened  an  international  assemblage  of  subject  matter  experts  in  objective  assessment  of  surgical 
technical  skills  and  of  representatives  of  numerous  official  bodies  involved  in  surgical  education, 
evaluation  and  certification.  There  have  been  recent  advances  in  the  educational  science  of  objective 
assessment  and  the  technology  of  medical  modeling  and  simulation  (MM&S),  using  surrogate  tissue, 
abstract  objects  or  virtual  reality  (VR)  systems,  for  training  of  technical  skills.  In  addition,  focus  on 
identification  and  prevention  of  medical  errors,  the  need  for  objective  criteria  for  assessment  of 
surgical  skills  and  the  increasing  demand  for  accountability  to  the  public  has  revealed  that  there  is  no 
infrastructure  for  the  objective  assessment  of  technical  skills.  Numerous  investigators  are  validating 
many  different  systems  for  training  and  evaluation,  using  different  tests,  criteria,  validation  methods 
and  even  nomenclature.  This  workshop  is  an  attempt  to  establish  a  standardization  of  nomenclature 
and  assessment  methodologies  so  the  surgical  education,  training  and  evaluation  community  can 
communicate  with  a  common  language  and  have  a  common  basis  to  compare  statistical  results.  The 
results  of  this  workshop  are  to  be  considered  a  first  order  approximation  from  the  community  of 
subject  matter  experts  that  can  provide  a  “strawman”  for  future  refinement. 

Introduction  (Satava). 

The  introduction  provided  a  background  on  the  purpose,  goals,  objectives  and  timelines  for 
the  conference.  The  goals  (Table  1)  were:  1.)  to  review  currently  available,  peer-reviewed,  surgical 
simulation  systems  for  objective  assessment  of  technical  skills.  2.)  To  agree  upon  a  set  of 
definitions  which  would  form  the  basis  of  discussion  for  the  workshop,  but  to  also  establish  an  agreed 
upon  lexicon  of  terms  which  all  the  current  researchers  in  objective  assessment  would  adhere  to,  and 
set  a  standard  of  definitions  in  objective  assessment  for  future  investigators.  3.)  To  develop  a 
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taxonomy  of  the  various  metrics  (abilities,  skills,  tasks  and  procedures)  that  comprise  the  “toolkit” 
which  investigators  can  apply  for  objective  assessment.  The  baseline  reference  toolkit  was  derived 
from  the  currently  accepted  and  scientifically  validated  instruments  which  have  been  used  for 
decades  by  researchers  in  behavioral  psychology,  ergonomics  and  human  interface  technology.  4.) 
To  match  the  metrics  in  the  taxonomy  (from  goal  3)  to  the  current  systems  (from  goal  1)  to  form  a 
matrix  or  “toolkit  of  objective  assessment  tools”  that  could  be  made  available  to  surgical  educators 
and  evaluators.  5.)  To  analyze  the  tools  (from  goal  4)  and  propose  a  generic  “core  curriculum”  that 
can  be  reviewed  by  the  international  community  of  surgical  educators  to  be  implemented  as  a 
standard  for  the  training  and  evaluation  of  surgical  technical  skills. 


Table  1.  Goals  of  Workshop 
Review  current  available  systems 
Definition  of  what  is  being  measured 
Develop  a  taxonomy  for  measurement 
Match  metrics  to  current  systems 
Develop  a  core  curriculum 


In  evaluating  the  currently  available  systems  of  simulators,  the  participants  were  specifically 
directed  to  assessing  the  validity  and  reliability  of  the  data  presented.  These  were  defined  based  upon 
definitions  from  a  standard  dictionary  (New  Webster’s  Dictionary  of  the  English  Language  (Deluxe 
Encyclopedic  Edition),  Delaire  Publishing,  Inc,  NYC;  1981)  and  a  dictionary  specific  to  the 
behavioral  psychology  profession  (Reber,  AS  Dictionary  of  Psychology )  (Appendix).  The  types  of 
validity  included  face,  content,  construct,  concurrent  and  predictive  validity  and  the  types  of 
reliability  included  inter-rater  and  test-retest  reliability  (Table  2). 

Finally,  the  timelines  for  the  accomplishments  of  the  workshop  were  delineated.  This 
included  the  completion  of  the  first  draft,  circulation  among  participants  for  review,  completion  of 
final  draft,  circulation  among  selected  organizations,  societies  and  regulatory  bodies  for  comment, 
and  for  preparing  the  final  report.  In  addition  there  were  to  be  plans  for  a  subsequent  open  forum  to 
insure  review  and  commentary  by  the  international  surgical  education  community  as  a  whole,  as  well 
as  an  advisory  board  to  insure  continuation  of  the  process  of  final  report  and  Open  Forum  planning. 
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Table  2  Validity  and  Reliability 
Validity 
Face 
Content 
Construct 
Concurrent 
Predictive 
Reliability 
Test-retest 
Inter-rater 


Military  Medical  Modeling  and  Simulation  Program  (Magee) 

The  US  Army  Medical  Research  and  Material  Command  (USAMRMC)  Telemedicine  and 
Advanced  Technology  Research  Center  (TATRC)  has  several  initiatives  in  surgical  simulation  as  part 
of  a  broader  Medical  Modeling  &  Simulation  (MM&S)  portfolio.  The  main  initiative  in  surgical 
simulation  is  a  Congressionally  Special  Interest  program  with  significant  funding  to  implement 
simulation-based  training  for  military  medical  requirements.  TATRC’  program  scope  is  broader  than 
simulation,  with  a  vision  to  identify,  explore,  and  demonstrate  key  technologies  and  biomedical 
principles  required  to  overcome  technology  barriers  that  are  both  medically  and  militarily  unique. 

In  April  1998,  the  General  Accounting  Office  (GAO)  reported,  “Military  medical  personnel 
have  almost  no  chance  during  peacetime  to  practice  battlefield  trauma  care  skills.  As  a  result, 
physicians  both  within  and  outside  the  Department  of  Defense  (DOD)  believe  that  military  medical 
personnel  are  not  prepared  to  provide  trauma  care  to  the  severely  injured  soldiers  in  wartime. . ..” 
(GAO/NSIAD-98-75).  With  some  of  today’s  training  methods  disappearing,  the  challenge  of 
providing  both  initial  and  sustainment  training  for  almost  100,000  military  medical  personnel  is 
becoming  insurmountable.  The  “training  gap”  is  huge,  and  impediments  to  training  are  mounting. 
This  led  the  TATRC  to  identify  needs  for  simulation  (Table  3). 

Thus  the  military  sees  a  unique  need  to  invest  in  these  training  modalities  and  are  required  to  go 
forward  regardless  of  civilian  pursuits.  The  military  has  therefore  already  established  a  Medical 
Modeling  and  Simulation  (MM&S)  initiative  and  assumed  a  leadership  role  in  surgical  training.  In 
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addition,  the  “To  Err  is  Human”  report  (Kohn,  Corrigan,  Donaldson,  Institute  of  Medicine. 
November,  1999)  has  focused  the  public’s  attention  on  the  issue  of  medical  errors  and  added  a  strong 


Table  3.  Identified  needs  for  simulation 

•  Training  should  replicate  combat  specific  wounds  &  battlefield  environmental  stressors. 

•  Civilian  injuries  are  unlike  war. 

•  Traditional  “see  one,  do  one,  teach  one”  method  is  changing. 

•  There  are  increased  restrictions  on  animal  use  for  training. 

•  Cost  to  conduct  mass  casualty  exercises  is  prohibitive. 

•  Health  care  payors  are  resistant  to  reimburse  for  training. 

•  There  are  risks  from  treatment  by  care  providers-in-training. 


civilian  impetus  for  the  TATRC  program.  Of  all  errors,  those  related  to  surgical  procedures  are  the 
most  dramatic  and  media-attention  attractors.  The  Center  for  Patient  Safety  of  the  Agency  for 
Healthcare  Research  and  Quality  has  identified  simulation  technologies  as  an  opportunity  to  increase 
patient  safety  through  reduced  errors  by  better  surgical  training. 

Based  upon  these  military  requirements  and  overall  public  awareness  of  the  medical  errors 
problem,  TATRC  has  developed  an  initial  strategy  which  includes  the  following  steps: 


Table  4.  Initial  Strategy 

Assess  the  landscape  -  perform  a  “Meta- Analysis” 

Engage  the  experts  -  implement  a  70-person  “Integrated  Research  Team” 

Converge  the  worlds-  bring  expertise  of  medicine  &  simulation  together 
Support  the  science  -  integrate  existing  congressional  and  other  funding  sources 
Designate  a  “lead  agency”  -  integrate  efforts  (TATRC  has  been  designated) 

Be  a  good  partner  -  develop  &  honor  professional  business  practices  at  all  stages  of  the  process 


The  thrust  of  TATRC’s  MM&S  portfolio  is  an  integrated,  multi-level  program  (figure  1).  It  takes 
into  consideration  the  multidisciplinary  approach,  involving  users  at  all  levels  and  in  all  agencies  and 
bringing  multiple  sources  of  funding  into  a  rich  portfolio  of  projects  that  support  the  overall  program 
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goals.  This  results  in  a  very  robust  program  that  can  leverage  off  many  different  disciplines  to 
support  MM&S  in  the  many  components,  from  partial  task  trainers  to  scenario  based  systems. 


PORTFOLIO 


COLLABORATIONS 


Figure  1  The  TATRC  Medical  Modeling  and  Simulation  Program 


TATRC  has  categorized  its  portfolio  of  simulators  into  4  categories  to  meet  the  military  needs. 
They  are:  1.)  Personal  computer  (PC)  based  interactive  multimedia,  2.)  Digitally  enhanced 
mannequins,  3.)  Virtual  workbenches,  and  4.)  Total  immersion  virtual  reality.  Each  type  of 
simulator  has  strengths  and  weaknesses,  and  the  development  of  a  core  curriculum  will  need  to  take 
the  strengths  of  each  to  form  an  optimal  educational  strategy. 

The  TATRC  program  will  develop  effective  medical  training  simulators,  by  developing  the 
fundamental  science  and  enabling  technologies  that  will  permit  learning  on  a  simulator  to  be 
translated  to  actual  patient  care.  Once  this  knowledge  is  available,  developers  can  create  application- 
specific  learning  modules,  from  first  responder  trauma  triage  through  surgical  part-task  simulators. 

A  number  of  research  challenges  have  been  identified  by  TATRC  during  the  initial  Integrated 
Research  Team  (IRT)  workshop  on  modeling  and  simulation.  These  include  (Table  5): 
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Table  5.  Research  Challenges 

Real-time  in  vivo  tissue  property  measurement  and  mathematical  modeling 

Tissue-tool  interactions,  haptics 

Graphics  and  visualization 

Learning  systems 

Validation  and  metrics 

Open  source  architecture  (Common  Anatomical  Modeling  Language  -  CAML) 


There  are  key  components  to  the  program  management,  including  (Table  6): 


Table  6.  Key  Components 

•  Expand  the  consortia  to  address  the  core  problems  facing  simulation. 

•  Constructively  collaborate  on  various  aspects  of  essential  research. 

•  Support  workshops  in  key  areas  of  science. 

•  Demonstrate  that  simulator-based  learning  actually  transfers  to  patient  care,  through 
validation  studies  and  development  of  training  metrics  based  upon  simulator  use. 

•  Identity  sufficient  funds  to  answer  these  needs  within  the  next  five  years. 


The  program  will  be  based  upon  robust  validation  methodology,  with  the  first  application  being 
the  Combat  Trauma  Patient  Simulation  (CTPS)  System,  a  system  of  digitized  mannequins  -  with 
Independent  Test  &  Evaluation  beginning  July,  2001. 

Oversight  is  conducted  by  refining  the  integrated,  tech-based  investment  strategy  for  MM&S  in 
direct  support  of  Joint  Warfighter  Science  &  Technology  Plan  through  the  establishment  of  working 
groups  for  functional  proponency,  technology  proponency  and  integration.  The  program 
management  includes  establishing  Integrated  Product  Teams  (IPT)  for  managing  specific  projects, 
such  as  Combat  Trauma  Patient  Simulator  (CTPS),  Simulation  Technologies  for  Advanced  Trauma 
Care  (STATCARE),  and  the  Medical  Simulation  Trainer  Initiative  (MSTI)  and  then  receiving, 
evaluating,  responding  to  scientific  proposals  and  workshop  requests... and  funding  some  of  them. 
The  USAMRMC  Broad  Agency  Announcement  Number  99-1  is  an  available  avenue  through  which 
interested  parties  may  submit  research  proposals. 
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In  summary,  the  military  is  moving  aggressively  in  developing  and  integrating  simulation  and 
objective  assessment  into  the  training  of  all  combat  medics,  nurses,  physicians  and  surgeons. 
Programs  are  identified,  funded  and  provided  oversight.  Once  appropriate  progress  has  validated  the 
systems,  the  military  will  be  in  position  to  be  the  leaders  in  simulation  for  medical  technical  skills 
training  and  evaluation. 

Goal  1:  REVIEW  OF  CURRENTLY  AVAILABLE  SYSTEMS 

Master  Surgeons  and  Surgical  Proficiency  (Cuschieri) 

The  issue  of  training  surgeons  revolves  around  determining  surgical  competence.  There  are 
two  basic  types  of  operative  surgeons  -  the  innately  unbelievable  operators  (IUO)  and  the  acquired 
master  craftsman  (AMC).  The  proficiency  of  these  surgeons  is  demonstrated  by  their  surgical 
operative  proficiency,  which  is  a  combination  of  the  efficiency  and  quality  of  their  task  performance 
(surgical  operative  proficiency  =  task  efficiency  x  task  quality).  Performance  can  be  in  either  an 
autonomous  “subthalamic”  mode  or  a  cognitive  “cortical”  mode.  In  addition  performance  must  take 
into  account  the  learning  curve,  the  rate  at  which  a  surgeon  attains  proficiency. 

Surgical  competence  is  achieved  by  selecting  the  right  people  and  placing  them  in  appropriate 
training/assessment  programs  which  include  the  full  range  of  competencies  needed.  Once  attained, 
there  is  the  need  for  both  continued  professional  education  and  re-validation.  However,  this  must  be 
obtained  using  a  new  paradigm;  no  longer  should  there  be  so  much  audit  with  it’s  putative 
implications,  rather  it  should  be  based  upon  quality  performance  to  bring  the  surgeon’s  proficiency 
into  a  region  which  reduces  risk  and  error  into  the  “as  low  as  reasonably  possible”  (AL ARP)  region. 
Some  surgeons  will  immediately  rise  to  a  level  of  proficiency,  the  majority  will  remain  in  the 
“learning  zone”  for  a  longer  time  until  proficiency  is  obtained,  and  a  few  will  never  attain  a  true  level 
of  proficiency  (figure  2). 

Current  training  in  surgery  spans  the  full  spectrum,  from  selection  of  trainees  based  upon 
aptitude  through  clinical  apprenticeship,  using  skills  courses  (including  simulators)  and  training 
initiative  lists.  However,  this  traditional  methodology  is  no  longer  sustainable;  for  example,  only 
20%  of  United  Kingdom  consultants  currently  supervise  colon  resections  (Aitken,  et  all,  1999).  An 
estimate  of  the  order  of  magnitude  of  effort  to  raise  the  supervision  of  trainees  from  30%  to  70% 
supervision  would  require  an  additional  270  operating  theater  days  in  the  United  Kingdom  at  a 
minimal  cost  of  £1.2  million  pounds  (£2million)  in  addition  to  the  numerous  extra  teaching  sessions. 
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Figure  2.  Learning  curves  and  levels  of  proficiency 


It  is  imperative  for  academic  surgeons  to  examine  the  different  levels  of  competence  in  order  to 
reach  decisions  about  competence  and  proficiency,  including  abilities,  skills  and  tasks.  Abilities 
(innate  attributes  for  surgery)  include  cognitive  ability,  personality  traits  and  psychomotor  ability. 
Although  cognitive  and  personality  traits  are  important  for  competence,  technical  skills  revolve 
around  psychomotor  abilities.  There  are  non-motor  abilities  (spatial  and  verbal)  and  motor  abilities 
(control  precision,  two-hand  coordination,  hand-arm  steadiness,  eye-hand  coordination  speed,  aiming 
and  manual  dexterity).  Among  spatial  abilities,  there  are  a  number  of  sub  classifications  ( visio- 
spatial,  spatial  visualization  and  spatial  orientation),  all  of  which  correlate  to  surgical  skills. 

There  are  numerous  factors  which  also  must  be  taken  into  consideration.  These  include:  1.) 
adaptability  (the  extent  to  which  a  person  is  able  to  interact  effectively  and  appropriately  with  his  or 
her  physical  and  social  environment  -  Aiken,  1994),  which  translated  to  the  surgical  skills  arena 
implies  the  ability  to  acquire  new  skills.  2.)  Perceptual  processing  of  indirect  images  (The  acquisition 
and  processing  of  sensory  information  in  order  to  see  objects  of  indirect  images;  also  guides  an 
individual’s  action  with  respect  to  these  objects)  and  perceptual  speed  (the  ability  to  recognize 
similarities  and  differences  quickly  -  Schofield,  1972).  3.)  Perceptual  motor  coordination  (The 
ability  to  generate  appropriate  muscular  commands  so  that  our  limbs  reach  position  in  space  specified 
by  our  perceptual  system,  for  example  eye-hand  co-ordination  -  Eyesenck,1994),  4.)  Multi-limb 
coordination  (The  ability  to  co-ordinate  the  movements  of  a  number  of  limbs  simultaneously  - 
Fleishman,  1966).  5.)  Aiming  (The  ability  to  quickly  and  precisely  perform  a  series  of  movements 
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requiring  eye-hand  co-ordination  -  Fleishman,  1966),  6.)  Arm-hand  steadiness  (The  ability  to  make 
precise  arm-hand  positioning  movements  where  strength  and  speed  are  minimized;  the  critical  factor 
being  the  steadiness  with  which  such  movements  can  be  made  -  Fleishman,  1966)  and  7.)  Control 
precision  (Common  to  tasks  which  require  fine,  highly  controlled,  but  not  over-controlled  muscular 
adjustments,  primarily  where  large  muscle  groups  are  involved). 

During  a  consensus  conference  of  master  surgeons  deliberating  the  importance  of  attributes  which 
bear  weight  on  the  determination  of  competence,  it  was  confirmed  that  cognitive  factors  and 
personality  traits  were  of  the  highest  importance,  while  dexterity  was  perceived  only  as  important, 
and  research  output  received  much  lower  valuation  (figure  3).  However,  when  the  evaluation  group 
considered  the  relative  impact  of  the  various  factors  upon  technical  competence,  innate  dexterity  was 
clearly  the  most  important  determinant,  while  both  cognitive  factors  and  personality  traits  were 
considered  less  important. 

Finally,  it  is  essential  to  bring  into  the  equation  the  importance  of  the  feedback  which  is  given  to 
the  student  during  the  training  episodes,  which  enhances  the  acquisition  of  skills.  Feedback  can  be 
concurrent  (real  time)  or  terminal  (after  completion).  With  concurrent  (or  “proximate  learning”),  the 
subject  is  made  aware  of  the  error  and  can  immediately  correct  it,  whereas  with  terminal,  there  is  an 
external  assessment  of  the  performance. 
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Figure  4  Impact  of  attributes  on  technical  competence 

Current  Simulators 


ADEPT 


(Advanced  Dundee  Endoscopic  Psychomotor  Tester) 
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The  ADEPT  is  a  computer  based,  microprocessor  controlled  goniometer  system  with  real  time 
recording  of  x,y,z,  position  of  active  and  passive  instruments  during  bimanual  dexterity 
manipulations.  Plate  exercises  provide  task  targets  and  animal  exercises  provide  surgical  tasks. 

In  addition  the  system  has  3  infra-red  cameras  which  locate/track  reflective  markers  and  tele- 
electromyographic  components  which  transfer  data  to  an  on-line  computer  where  the  system 
software  analyses  motion  patterns  in  3-D.  (figures  5,6).  The  system  is  a  standardized,  objective 
and  reliable  assessment  method  for  endoscopic  training  because  it  discriminates  between 
individuals  in  terms  of  their  innate  abilities  to  perform  endoscopic  tasks.  The  system  has  been 
assessed  for  reliability  (Arch  Surg,  1999)  and  has  been  shown  to  have  test-retest  reliability  and  to 
abolish  inter-rater  variability  with  internal  consistency  (coefficient  alpha  =  0.97  for  inter-tasks 
and  intra-tasks).  Neither  significant  improvement  with  practice  (p>0.1  and  correlation  coefficient 
of  0.64  between  runs)  nor  discriminative  ability  (significant  difference  between  subjects  = 
pO.OOl  and  coefficient  of  variation  =  68%  for  error)  are  evident. 
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Figure  6.  ADEPT  showing  internal  components 


Face  validity  of  ADEPT  has  been  demonstrated  (Surg  Endosc,  1998)  by  using  standard 
endoscopic  instruments  and  video  endoscopic  system.  The  instrument  movement  has  the  same 
degrees  of  freedom  as  through  an  access  port  and  the  task  plate  reproduces  generic  endoscopic 
movement.  In  addition  real  surgical  tasks  can  be  used,  e.g.  knot  tying  and  bowel  anastomosis  by 
replacing  the  plate  with  simulants  or  tissues.  Concurrent  validity  (correlates  with  Crawford  small 
part  dexterity  test  -  ESSR,  Malmo  2000)  has  been  established.  Construct  validity  is  proven  because  it 
differentiates  between  trainees  and  expert  surgeons  (EAES,  Nice  2000)  and  predictive  validity  is 
demonstrated  by  the  correlation  between  clinical  competence  amongst  trainees,  r  =  0.7  (Am  J  Surg 
1998). 

Motion  analysis  is  used  to  compare  specific  task  performance  between  experts  and  trainees. 
ADEPT  is  able  to  establish  what  the  experts  do  differently  than  trainees  for  a  given  task.  Trainees 
simulate  the  experts’  movements  to  improve  task  efficiency  &  quality.  Task  efficiency  at  knot  tying 
reveals  that  experts  average  60  seconds,  while  trainees  average  120  sec  (p<0.0001  by  Mann- Whitney 
U-test),  higher  quality  of  knots  with  experts  at  35%  and  trainee  at  2%  (p<0.0005  by  Mann- Whitney 
U-test)  and  efficiency  by  angular  velocity  of  the  shoulder  and  elbow  (p<0.05  by  Mann- Whitney  U- 
test)  and  reduction  of  abduction  in  range  of  motion  ((p<0.05  by  Mann- Whitney  U-test). 
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Figure  7  Motion  analysis  with  reflectors  and  electromyographic  electrodes. 

Laparoscopy  Cholecystectomy  Simulator  (Xitact,Inc) 

The  Laparoscopic  Cholecystectomy  Simulator  (figure  8)  by  Xitact,  Inc  is  a  hybrid  mannequin 
and  virtual  reality  system  with  a  mechanical  haptic  input  component  (using  Stewart  platforms  and 
optical  encoders)  to  provide  force  feedback  and  an  image  (figure  9)  which  is  computer  generated. 


Figure  8  Xitact  -  external  view  Figure  9  Xitact  -  computer  image 

The  system  is  similar  to  many  of  the  previous  systems  developed  to  train  and  evaluate  the  entire 
procedure  of  laparoscopic  cholecystectomy.  An  objective  assessment  system  is  currently  under 
development. 

MISSIMU  (Minimally  Invasive  Surgery  Simulator)  Dundee  prototype 

The  European  MISSIMU  (Minimally  Invasive  Surgery  Simulator),  which  has  been  developed 
by  a  European  consortium,  is  also  a  hybrid  simulator  (figure  10)  and  represents  the  latest  in 
mechatronic  (combined  mechanical  and  electronic)  systems  to  provide  the  force  feedback  and  a  high 
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fidelity  image  (figure  11).  The  system  has  taken  to  a  new  level  the  fidelity  in  haptic  input  by 
devising  a  new  mechatronic  system  (figure  12).  The  system  has  been  devised  to  enable  both  the 
virtual  reality  system,  as  well  as  the  opportunity  to  use  simulants  under  the  dome  of  the  mechatronic 
device. 


Figure  10.  Two  external  views  of  the  MISSIMU  simulator 


Figure  11.  High  fidelity  image  of  the  simulator  Figure  12  Haptic  input  device 


The  simulator  continues  undergoing  evaluation  with  a  comprehensive  assessment  system  is 
continuing  development.  No  validation  data  have  been  released. 

Performance  Levels 

In  reviewing  the  literature  and  convening  the  consensus  of  master  surgeons,  an  ontology  of 
performance  levels  can  be  constructed  which  reflects  both  the  level  of  training  and  overall  technical 
competence.  The  proposed  levels,  based  upon  those  proposed  by  Stuart  and  Hubert  Dreyfus,  are: 
Novice,  Competent,  Proficient,  Expert  and  Master  (Appendix  7). 
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ADEPT  —  The  Eindhoven  E.A.E.S.  Study  (Jakimowicz) 

The  Advanced  Dundee  Endoscopic  Psychomotor  Tester  (ADEPT)  system  was  evaluated 
during  the  9th  International  Congress  of  the  European  Association  of  Endoscopic  Surgeons,  13-16 
June,  200 1 .  There  were  45  laparoscopic  surgeons  who  participated,  each  receiving  an  instructional 
trial  and  a  test  trial  (60  seconds  per  task).  A  post-test  Visual  Analogue  Scale  (VAS)  questionnaire 
measuring  attitude  towards  skills  testing  in  general  validation  and  performance  on  ADEPT  was  used. 
The  outcomes  denominators  were  hand-to-eye  coordination,  spatial  perception,  two-handed 
coordination,  arm-hand  steadiness  and  manual  dexterity.  The  following  hypotheses  were  tested. 

Is  performance  on  ADEPT  reflecting  innate  psychomotor  ability?  (Scores  will  not  improve 
with  succeeding  trials,  there  is  no  learning  curve.) 

Participants’  scores  on  their  test-run  and  true-run  were  compared.  Patricipants  did  improve  in 
total  time  needed  for  runs  (test  run  mean  time:  129  +  54.4  seconds  versus  true  run  mean  time:  100  + 
39.3  seconds).  Also,  the  number  of  successful  task  performances  on  the  true  run  was  higher  than  on 
the  test  run  (2.4  +  1.3  versus  3.3  +  1.0  successes).  Innate  ability  is  established  as  surgeons’  score 
express  high  concordance  between  test-run  and  true-run,  as  72.2%  of  participants  expressed  a  true- 
run  score  within  on  distance  from  test-run.  On  paired  samples  Student’s  T-test,  both  time  and  score 
are  significantly  different  (p=0.001  for  both  variables)  indicative  for  some  improvement/training 
effect.  Thus  the  ADEPT  is  principally  measuring  fundamental  psychomotor  ability. 

Are  surgeons  good  estimators  of  their  own  performance  on  ADEPT  ?  (Scores  on  the  ADEPT 
will  correlate  with  their  estimation  of  their  ability  on  the  VAS) 

To  quantify  surgeons’  estimate  of  their  own  performance,  participants  were  asked  the 
following  question:  “If  I  had  to  give  myself  a  score  for  my  overall  performance  on  ADEPT,  this 

score  would  be _ ”.  This  subjective  perception  of  one’s  own  performance  was  compared  with 

the  objective  score  on  ADEPT.  If  surgeons  are  reliable  estimators  of  their  own  performance,  a  linear 
relationship  should  be  visible  when  both  scores  are  plotted  against  each  other.  In  order  to  do  so,  the 
authors  computed  a  sum  score  parameter  estimation  (SUM)  of  objective  individual  performance  on 
ADEPT.  This  variable  SUM  takes  the  following  into  account:  Successful  tasks  (0-5  points/run), 
total  execution  time  <150  seconds  (1  point),  number  of  “perfect  plate  tasks”  (1  point),  and  score  on 
test  run  with  at  least  one  task  successful  (1  point).  Thus,  the  maximum  score  of  10  could  be 
achieved.  SUM  approaches  the  normal  distribution  and  elements  in  SUM  were  in  concordance. 

SUM  was  then  plotted  against  subjective  performance  score.  Mean  VAS  score  for  the  test-run  was 
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5.4  (±  3.8)  and  mean  VAS  score  for  the  true-run  was  6.1(+  3.4).  Mean  score  on  SUM  is  (6.6  ±  1.64). 
In  general,  surgeons  tend  to  modest  in  grading  themselves  for  their  performance  on  ADEPT. 
Nonetheless,  the  VAS  score  does  not  seem  to  be  a  reliable  predictor  of  objective  performance,  since 
confidence  intervals  for  scores  of  SUM  all  seem  to  overlap. 

Do  surgeons  perceive  the  ADEPT  to  be  a  valid  instrument  in  measuring  laparoscopic  skills  ? 

In  general,  there  was  no  consensus  on  the  validity  of  the  system;  the  participants  were  equally 
divided  in  their  evaluation. 

Is  ADEPT  useful  for  validation  of  laparoscopic  training  techniques  ? 

The  system  actually  assesses  fundamental  abilities.  It  would  be  necessary  to  use  a  simulator 
which  tests  skills,  since  skills  have  a  learning  curve  (improve  over  time)  and  therefore  can  detect  the 
improvement  over  time  which  would  indicate  the  training  value.  The  next  protocol  for  the  ADEPT 
evaluation  will  include  the  following  (Table  7): 


Table  7  Next  protocol  for  the  ADEPT 

•  Cohort  of  residents-in-training  2001  four  regions  (N=40) 

•  T0-T6-T12-T18-T24  measurements  (months) 

•  One  extra  measurement  following  Basic  Surgical  Skills  Course  in  Leiden,  The  Netherlands  + 
instructed  observer  measurement  of  endoscopic  performance  on  VAS 

•  Outcomes  will  be  correlated  to  (previous)  surgical  and  endoscopic  experience 

•  Clinical  supervisor  will  be  asked  once  a  year  about  abilities/progress 

•  Residents  and  Clinical  Supervisor  will  not  know  outcomes  during  study 


Assessment  Tools  from  the  Imperial  College,  London  (Darzi) 

The  Imperial  College  School  of  Medicine,  London  has  developed  and  validated  several  objective 
measures  of  skill,  applied  to  both  virtual  reality,  inanimate  and  organic  simulation.  The  tools  developed 
are  the  Imperial  College  Surgical  Assessment  Device  (ICSAD)  and  the  Observer™  Pro  Video  Scoring 
System.  In  addition,  validation  has  been  performed  upon  the  Minimally  Invasive  Surgery  Trainer  - 
Virtual  Reality  (MIST-VR)  system  by  Mentice,  Inc  and  the  Royal  College  of  Surgeons’  Basic 
Surgical  Skills  curriculum  using  an  Objective  Structured  Clinical  Exam  (OCSE)  in  conjunction  with 
the  Observer™. 
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The  Imperial  College  Surgical  Assessment  Device  (ICSAD) 

The  ICSAD  is  a  combination  of  a  commercially  available  electromagnetic  motion  tracking 
system  (Polhemus  Inc,  figure  13)  which  tracks  instrument  movement  and  compiles  raw  x,y,z  data,  using 
a  Butterworth  filter  to  smooth  data,  which  is  noise-gated  and  time  stamped  (figure  14)  using  proprietary 
computer  software  developed  in  house  at  Imperial  College.  The  custom  software  extrapolates  this 
positional  data  into  scores  of  dexterity  and  movement  efficiency,  namely  the  number  of  movements 
made  by  the  hands,  pathlength  of  hand/  instrument  travel,  and  the  time  taken.  As  with  all  computer 
based  systems,  reliability  is  high  since  there  is  no  “inter-observer”  disagreement.  Current  limitations  to 
motion  analysis  include:  1.)  No  measure  of  quality  is  determined,  only  quantitative  data.  2.)  Only 
dexterity  is  assessed,  not  other  skills  like  visio-spatial,  etc  and  3.)  Extensive  outcome  analysis  research 
is  necessary  to  validate  long  term  outcomes. 

ICSAD  is  a  flexible,  reliable  assessment  tool  that  has  been  used  in  laparoscopic  simulation,  open 
surgical  simulation  and  within  the  operating  theatre  itself.  It  is  not  dependent  on  human  analysis  and 
interpretation,  meaning  that  it  is  both  truly  objective  and  labor  efficient.  The  ICSAD  has  been  used  to 
demonstrate  the  validity  of  many  models  and  simulations  to  show  both  construct,  concurrent  and 
predictive  validity,  as  well  as  reliability..  Examples  of  these  studies  are: 

Construct  validity 

ICSAD  has  been  shown  to  be  a  valid  assessor  and  discriminator  of  experience  in  a  number  of 
laparoscopic  and  open  surgical  models,  such  as  knot  tying,  suturing,  bowel  and  vascular  anastomosis.  In  all 
of  these,  experienced  surgeons  demonstrate  greater  movement  efficiency.  ICSAD  has  used  path  length  as  a 
measure  of  a  surgeon’s  experience.  There  is  an  inverse  relationship  between  the  path  length  and 
experience;  experienced  surgeons  have  learned  how  to  move  directly  (short  path  length)  to  perform 
procedures,  but  medical  students  or  those  with  no  laparoscopic  surgery  experience  demonstrate 
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Figure  13  ISCAD  Motion  Tracking  (Polhemus)  Figure  14.  Raw  data  of  motion  tracking,  before  filtering 


long  path  lengths,  (figure  15).  In  open  surgical  techniques,  the  teaching  of  mattress  sutures  reveals  that 
senior  house  officers  had  more  hand  movements  than  experienced  surgeons  (figure  16).  Thus  there  is  a 
strong  relationship  with  ISCAD  between  movements  made  and  time  taken,  however  this  relationship  is 
not  fixed.  When  further  correlation  tests  are  applied  in  all  these  experiments,  such  as  controlling  for 
time,  movements  still  correlate  with  experience  /  expertise  /  better  outcomes.  This  is  not  the  case  when 
controlling  for  movements,  which  suggests  that  time  taken  is  secondary  to  number  of  movements  made. 


Figure  15.  Path  length  as  a  function  of  experience 
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Figure  16.  Hand  movement  a  function  of  experience 
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Concurrent  validity. 

Performance  on  a  number  of  models  has  been  assessed  using  both  ICSAD  and  other  validated 
techniques,  such  as  OSATS.  Significant  correlation  between  these  two  methodologies  has  been  shown. 
Predictive  validity. 

Relative  performance  on  a  vascular  model  (anastomosis)  as  measured  with  ICSAD  versus 
outcome  (leakage  across  graft)  has  shown  a  significant  relationship  between  skill  and  outcome  (decreased 
leakage). 

ISCAD  has  also  been  used  to  demonstrate  the  efficiency  and  impact  of  new  technologies.  For 
example,  one  study  has  shown  the  benefits  of  a  3D  laparoscopic  camera  system  with  improved  movement 
efficiency.  Another  has  demonstrated  the  improved  skill  in  knot  tying  and  dissection  inferred  by  a 
laparoscopic  hand  assist  device. 

The  Observer™  system 

The  Observer™  is  a  video  digitization  system  that  is  able  to  be  programmed  to  quantify 
errors.  The  system  has  been  developed  by  psychologists  as  a  means  of  analyzing  seemingly  random 
human  interactions  in  an  objective  manner.  When  applied  to  surgical  simulation,  quality  of 
performance  can  be  assessed  without  the  need,  or  expense,  of  highly  trained  examiners.  Different 
elements  and  surgical  skills  can  be  assessed,  the  focus  dependent  on  what  a  particular  study  is 
concentrating  on  (eg:  errors,  order  of  operative  steps  etc).  In  addition,  the  system  is  computer  based, 
so  that  performance  which  has  been  digitized  can  be  analyzed  retrospectively  and  anonymously. 
Again,  this  is  a  flexible  analysis  system  that  be  applied  to  a  spectrum  of  models  and  simulation 
environments,  both  laparoscopic,  open  and  organic.  The  marriage  of  human  and  computer  based 
analysis  has  also  allowed  very  high  inter  examiner  reliability  to  be  demonstrated  (typically  over 
0.85),  lending  greater  confidence  in  both  test  task  and  assessment.  For  example,  a  study  to  compare 
and  contrast  performance  during  (porcine)  cholecystectomy  demonstrated  an  inter  rater  reliability 
between  four  observers  of  0.89.  Each  operation  was  recorded  and  digitized  (figure  17)  and  four 
independent  surgeon  observers  scored  each  cholecystectomy.  The  results  (Table  8)  clearly 
demonstrate  a  very  high  level  of  correlation,  indicating  that  Observer™  is  a  reliable  tool  for 
automatically  assessing  the  performance  of  tasks  that  are  recorded  on  video  tape. 
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The  Observer  -  Event  Recorder 
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Figure  17  Observer  automatic  assessment  system 
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*  Correlation  significant  at  the  0.01  level 


Table  8.  Results  of  correlation  between  the  Observer  system  and  the  surgeon  observers 

MIST-VR  (Minimally  Invasive  Surgery  Trainer  -  Virtual  Reality 

The  MIST-VR  simulator  is  a  personal  computer  (PC)  based  system  which  uses  laparoscopic 
graspers  and  a  computer  generated  image  of  abstract  objects  (figure  18).  The  tasks  are  based  on  the 
different  skills  needed  for  laparoscopic  surgery,  with  movement  efficiency,  path  length  and  errors 
recorded.  However  there  is  no  tactile  feedback.  Studies  at  Imperial  College  have  shown  construct 
validity,  i.e.:  the  system  can  differentiate  correctly  between  levels  of  expertise.  MIST  has  proved  to  be  a 
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useful  assessor  of  psychomotor  skill.  Further  work  has  looked  at  the  influence  of  environment  on  ability, 
for  example,  sleep  deprived  surgeons  show  greater  movement  inefficiency  and  errors  than  those  fully 
rested  (figure  19).  Studies  assessing  the  effects  of  alcohol  on  performance  have  revealed  similar  results. 


Figure  18  MIST-VR  system  Figure  19  Effect  of  sleep  deprivation  on  performance 


Competence  Day 

A  surgical  skills  assessment  package  for  basic  trainees  has  been  developed  and  validated  within 
the  institution.  The  system  is  based  on  a  six  station  OSCE  format  (figure  20),  using  synthetic  and  virtual 


Figure  20  Students  at  6  station  OSCE 

reality  simulation,  and  the  assessment  processes  outlined  above.  The  stations  have  been  chosen  to  reflect 
the  important  skills  that  trainees  should  have  gained  after  two  years  of  their  residency.  They  are: 
knowledge  of  instruments  and  sutures,  knot  tying,  suturing,  excision  of  sebaceous  cyst,  closure  of 
enterotomy,  and  basic  laparoscopic  navigation  and  manipulation.  All  stations  have  been  validated  with  a 


30 


Metrics  Conference 


I  August  2, 2001 
Version  1.3 


combination  of  two  assessment  tools  (ICSAD,  OSATS,  Observer,  MIST),  and  it  is  encouraging  that 
strong  correlations  between  these  tools  have  been  shown,  increasing  reliability  and  confidence  in  the 
process.  Construct  validity  has  been  shown  comparing  junior  trainee  performance  against  senior  trainees 
and  consultants. 

In  summary,  the  Imperial  College  has  implemented  a  comprehensive  training  and  evaluation 
system  which  includes  the  traditional  open  surgery  skills  to  the  state-of-the-are  in  motion  tracking  and 
virtual  reality.  The  systems  can  be  used  for  assessing  aptitude,  training  technical  skills,  evaluating 
performance  and  conducting  research  in  variable  conditions,  such  as  under  stress,  etc. 

Training  Center  Minimal  Invasive  Surgery  University  of  Tuebingen  (Buess) 

The  Univesity  of  Tuebingen  has  been  conducting  standardized  training  in  surgical  skills  for  over 
12  years.  The  focus  has  been  upon  minimally  invasive  procedures,  from  laparoscopic  surgery  to  Trans- 
anal  Endoscopic  Microsurgery  (TEMS).  Simulators  have  been  designed  which  are  mechanical 
phantoms  (figure  21)  as  well  as  animal  tissue  parts  mounted  into  moulds  of  the  human  torso,  such  as  the 
artificial  anus  for  TEMS  simulation  (figure  22). 


Figure  21.  The  phantom  for  TEMS  training  Figure  22  Example  of  tissue  based  simulator 

For  laparoscopic  surgery,  the  mechanical  system  (figure  23a)  has  been  developed  to  provide  the 
strengths  of  analysis  of  motion  with  computer  based  systems  in  evaluation  and  the  realism  of  using 
actual  animal  tissue  specimens  (figure  23b)  for  high  visual  and  tactile  fidelity. 
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An  assessment  of  technical  surgical  skills  has  been  performed  in  order  to  compare  different 
methods  of  training  with  objective  self-assessment  tools  that  measure  operative  skills,  observe  the 
learning  curve  and  eventually  provide  outcomes  analysis  that  will  allow  for  the  setting  of  standards.  The 
criterion  used  were:  Precision  (aiming  accuracy),  speed  (time  to  completion),  number  of  procedures, 
efficiency  of  movements  and  number  of  submovements.  A  structured  scoring  system  was  employed 
which  used  both  objective  measures  from  the  instruments  as  well  as  subjective  observer  evaluation.  In 
addition,  an  assessment  of  learning  progress  was  performed  on  an  abstract  psychomotor  analysis  model 
with  measurement  of  trajectories  by  ultrasound  triangulation  in  a  new,  portable  configuration  (figure 
24).  The  measured  values  are:  time,  x,y,z-  coordinates,  acceleration,  submovements,  detour-factor  and 
error  rate.  In  addition,  a  full  visualization  of  the  hand  motion  during  a  single  trial  can  be  created  using 
trajectory  analysis  (figure  25). 


Figure  24  Multiple  peg  tracking  task 


Figure  25  Trajectory  of  motion  with  20  target  points  touched 
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Analysis  of  the  hand  motion  velocity  as  a  function  of  time  (figure  26)  reveals  that  both  beginner  and 
expert  surgeons  have  an  initial  rapid  motion  toward  the  target,  but  beginners  waver  back  and  forth  trying 
to  touch  the  target,  while  experts  have  initial  velocity  which  deliberately  slows  down  and  quickly 
acquires  the  target. 


Beginner  (long  overall  duration,  multiple  velocity  peaks) 


Advanced  user  (one  peak,  then  careful  approach) 


Figure  26.  Results  of  motion  tracking  of  beginners  versus  expert  surgeons  in  multiple  peg  tracking  task. 

Comprehensive  objective  assessment  of  fundamental  abilities  for  laparoscopic 
surgery.  (Gallagher) 

The  introduction  of  laparoscopic  surgery  introduced  new  human  factors  and  ergonomic 
problems.  These  problems  include:  Loss  of  3-D  vision  (or  more  accurately  binocularity),  degraded 
tactile  cues,  degraded  visual  image,  inadequate  camera  etiquette  (e.g.  camera  rotation)  and  difficult  to 
leam  hand-eye  co-ordination,  especially  the  ‘Fulcrum  effect’. 

There  are  a  number  of  important  factors  that  impact  upon  performance  in  minimally  access 
surgery  (MAS).  These  include  cognitive  factors  (MAS  specific  knowledge)  and  technical  factors 
(psychomotor  ability,  visio-spatial  ability  and  perceptual  ability).  By  definition,  fundamental 
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psychometric  abilities  are  fixed  at  birth  or  early  childhood  and  show  little  or  no  learning  effect  (figure 
27).  These  difficulties  mean  that  the  minimal  access  surgeon  must  perform  surgery  operating  at  the  very 
edge  of  their  perceptual,  cognitive  and  psychomotor  abilities.  This  also  means  that  there  is  limited  room 
for  compensation  during  difficult  procedures. 
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□  Fundamental  psychological  abilities  fixed  at  birth  or  early  childhood  and  show 
little  or  no  learning  effects!  Focus  of  the  Belfast  efforts  on  objective  assessment. 


Figure  27.  Factors  affecting  minimally  access  surgery  performance 


Psychometric  abilities  can  be  assessed  using  the  Minimally  Invasive  Surgery  Trainer  -  Virtual 
Reality  (MIST-VR)  (figure  28).  This  system  (described  above)  is  a  psychomotor  assessment  device  of 
six  tasks  of  graded  complexity  which  require  two-handed  manipulation  of  virtual  spheres  and  cubes.  To 
date,  most  objective  research  on  MIST-VR  has  been  as  a  training  device,  although  it  may  be  that  MIST- 
VR  is  best  utilized  as  an  assessment  device.  The  system  uses  the  same  endoscopic  instruments  as  in  the 
operating  room  (OR)  requiring  the  two-handed  manipulation  of  virtual  objects  and  the  use  of  diathermy 
on  tasks  5  and  6.  The  limitations  include  no  representation  of  virtual  tissue  and  the  absence  of  the  sense 
of  touch  (haptics);  however  it  has  the  advantage  of  being  very  reliable  and  affordable. 

Initial  studies  focused  upon  a  psychomotor  ability  validation  of  MIST-VR  tasks  as  an  assessment 
device  with  a  pure  psychomotor  task  (concurrent  validity).  The  MIST-VR  was  correlated  with  a  pursuit 
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tracking  task  and  a  compensatory  tracking  task  (pure  psychomotor  ability).  The  pursuit  tracking  task 
requires  participants  to  track  a  randomly  moving  target  as  closely  as  possible  with  a  cross-hair;  the 
compensatory  tracking  task  requires  the  participants  to  use  a  joystick  to  attempt  to  keep  the  randomly 
moving  target  fixed  to  a  central  point.  The  results  show  a  high  level  of  correlation  (figure  29)  with  r 
values  of  0.792  or  greater. 


Figure  28  MIST -VR 
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Tracking 

scores 

Time  On  Target 

Factors  correlated 

R-value 

P-level 

R-value 

P-level 

Pursuit  with  MIST  VR 

0.792 

0.0222 

0.857 

0.0038 

Compensatory  with 

MIST  VR 

0.89 

0.0009 

0.89 

0.0009 

Figure  29  Results  of  pure  psychomotor  tracking  validation  task  with  MIST  VR  performance  (n  =  20) 

The  validity  of  the  MIST-VR  was  conducted  using  the  following  four  parameters:  Time  (the 
time  subjects  spent  from  when  they  started  on  the  task  until  the  time  that  they  completed  the  last 
sequence  of  movements),  Error  (average  error  was  measured  as  the  number  of  errors  per  task  segment.), 
Economy  of  Movement  (assessed  for  the  left  and  right  instrument  as  the  proportion  of  the  distance 
travelled  by  the  left  (or  right)  each  instrument  tip  (or  working  end)  that  has  exceeded  it’s  optimal 
distance)  and  Economy  of  Diathermy  (the  total  bum  time  was  used  as  a  measure  of  economy  of 
diathermy  score  for  both  tasks  5  and  6  only) 

Constmct  validity  was  demonstrated  between  experienced  surgeons  (i.e.,  >50  MAS)  versus 
junior  surgeons  (ie.,  <10  MAS).  The  experienced  surgeons  performed  MIST  VR  significantly  faster 
with  fewer  errors,  greater  economy  of  instrument  movement,  greater  economy  of  diathermy  and  greater 
consistency  (i.e.,  lower  SD’s).  MIST-VR  distinguishes  between  surgeons  of  different  levels  of 


36 


*  Metrics  Conference 


1  August  2, 2001 
Version  1.3 


experience,  has  a  high  test-re-test  reliability  r  =  0.5  -  0.93,  has  high  internal  validity  a  =  0.89  -  0.98,  has 
good  construct  validity  and  has  good  discriminative  validity  for  surgeons  of  different  levels  of 
performance  and  laparoscopic  novices  performance.  In  addition  the  process  of  data  collection  is 
automated. 

Perceptual  ability  is  assessed  using  the  Pictorial  Surface  Orientation  (PicSOr)  test  battery 
(Cowie,  Perception,  1998 )  which  is  designed  to  assess  the  ability  to  recover  3D  information  from  2D 
visual  cues  (figure  30).  The  test  is  specifically  designed  to  provide  only  those  cues  (with  shading  and  a 
rotating  “arrow”)  which  requires  judgment  of  depth  of  field. 
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Figure  30  PicSOr  Perceptual  Assessment  test  Figure  3 1  Visio-spatial  Card  Rotation  test 

Visio-spatial  ability  was  assessed  using  the  Manual  for  Kit  of  Factor-Referenced  Cognitive 
Tests.  (Ekstrom  et  al.,  1976).  The  specific  tests  used  were  Card  Rotation  Test  (spatial  orientation), 
Cube  Comparison  Test  (spatial  orientation),  Maze  Tracing  Test,  Choosing  a  Path  Test,  Map  Planning 
Test  (spatial  navigation),  Form  Board  Test,  Paper  Folding  Test  and  Surface  Development  Test.  The 
three  tests  which  correlated  to  a  laparoscopic  cutting  task  and  to  the  MIST-VR  were  Card  Rotation, 
Cube  Comparison  and  Map  Plan  (figure  32). 
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All  of  the  above  tests  correlate  well  with  the  MIST-VR  and  assess  principally  the  fundamental 
abilities.  Future  tests  need  to  determine  if  there  are  correlation  with  skills,  such  as  Test  5  and  Test  6  on 
MIST-VR.  These  tests  involve  the  use  of  diathermy  with  the  two  handed  manipulation  of  spheres  and 
cubes,  and  therefore  require  multiple  abilities  and  should  reflect  skills  which  can  be  learned. 


Cutting  task  MIST-VR  Time  MIST-VR  Error 


Visio  Spatial  Test 

r 

p-level 

r 

p-level 

r 

p-level 

Card  Rotation 

0.71 

0.001 

-0.73 

0.0005 

-0.67 

0.002 

Cube  comparison 

0.68 

0.002 

-0.78 

0.0001 

-0.69 

0.002 

Map  Plan 

0.45 

n.s 

-0.7 

0.0001 

-0.58 

0.013 

Figure  32.  Correlation  coefficients  and  their  associated  p-values  between  the  visio  spatial  test  scores 
and  the  three  different  measures  of  laparoscopic  performance  (n  =  20). 

The  complete  assessment  of  surgical  competence  and  MAS  performance  is  much  more 
comprehensive  than  simple  technical  ability.  Figure  33  demonstrates  the  many  other  components,  such 
as  experience,  physiologic  constraints  (fatigue,  stress,  etc),  equipment  constraints,  personality, 
teamwork,  etc.  that  must  be  factored  into  the  equation  of  competence  and  performance  assessment. 
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Figure  33  Multiple  factors  that  impact  upon  MAS  performance. 


The  conclusions  are  that  objective  assessment  of  skills  for  MAS  is  now  possible.  The  factors 
demonstrated  above  which  were  believed  to  be  related  to  MAS  performance  are  strongly,  and 
statistically,  significantly.  Virtual  reality  may  be  one  of  our  most  valuable  tools  in  this  enterprise  to 
provide  objective  measures  of  fundamental  abilities.  Now  it  is  necessary  to  establish  norms  for  these 
measures  with  national  (but  preferably  international)  performance  criterion  levels  for  fundamental 
abilities  for  MAS.  Lastly,  and  most  importantly,  it  is  necessary  to  determine  if  the  psychometrics  tests 
predict  skills  transference  and  OR  performance.  Evidence  from  other  high  skilled  professions  indicates 
that  they  will. 


The  Intercollegiate  Basic  Surgical  Skills  Course  (Royal  Australasian  College  of 
Surgeons)  (Deane,  Hamdorf) 

The  Royal  Australasian  College  of  Surgeons  (RACS)  conducts  surgical  training  across 
Australia  and  New  Zealand.  The  RACS  training  programs  extend  across  the  entire  spectrum  of 
training  and  include  selection  out  of  internship  (PGY1),  basic  surgical  training  (BST)  in  PGY2,3  with 
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possible  extension  into  PGY4,5,  advanced  surgical  training  (AST)  for  nine  major  specialties  and 
continuing  professional  development. 

The  BST  begins  with  selection  and  registration  for  training.  There  is  a  defined  syllabus  and 
compulsory  distance  learning  program  with  text  modules  and  interactive  on-line  resource  units. 
There  is  the  opportunity  for  a  distance  learning  programme  with  text  modules  and  interactive  on-line 
resource  units.  For  certain  circumstances  there  is  a  mentoring  programme.  Skills  courses  include 
Basic  Surgical  Skills  (BSS),  Early  Management  of  Severe  Trauma  (EMST  -  the  RACS 
implementation  of  ATLS)  and  Care  of  the  Critically  Ill  Surgical  Patient  (CCrISP).  EMST  began  in 
1988  by  arrangement  with  the  American  College  of  Surgeons.  CCrISP  began  in  2000  by 
arrangement  with  the  Royal  College  of  Surgeons  of  England.  BSS  is  the  intercollegiate  Basic 
Surgical  Skills  course  conducted  by  the  Surgical  Colleges  in  the  United  Kingdom  and  began  in 
Australia  and  New  Zealand  in  1999  by  arrangement  with  the  Royal  College  of  Surgeons  of  England. 
Each  of  these  courses  is  conducted  at  multiple  locations.  BST  evaluation  involves  both  formative 
and  summative  assessments,  including  a  clinical  skills  OSCE  and  a  pre-determined  standard  must  be 
reached  to  allow  application  for  AST.  The  RACS  is  establishing  collaborations  with  universities, 
state  governments  and  health  administrations,  other  professional  organizations,  and  private  industry 
to  set  in  place  the  necessary  facilities  and  resources  which  will  be  required  for  the  future  of 
laboratory-based  skills  training  in  Australia  and  New  Zealand. 

The  Intercollegiate  Basic  Skills  Course  has  been  described  (Hamdorf).  The  important 
components  of  the  Course  included  a  high  tutor  to  participant  ratio  (1:2  to  1:3),  fostering  an  air  of 
stewardship  and  team  approach,  allowing  an  opportunity  for  reflection.  The  course  is  delivered  at 
multiple  sites  with  varying  levels  of  technological  backup  throughout  Australasia  so  is  necessarily 
“low  tech”  in  its  approach. 

The  skills  are  taught  on  a  3  day  course  with  open  surgery  (1.5  days),  mulculoskeletal  injuries 
(0.9  days)  and  minimal  access  surgery  (1 .0  days)  and  use  operative  techniques.  The  tasks  performed  are 
listed  in  Figure  34. 

Open  surgical  skills  are  assessed  with  surrogate  tissues  or  animal  tissues.  Knot  tying  and 
abdominal  wound  closure  use  synthetics  (figure  35,  36),  whereas  excision  of  skin  lesions  and 
anastomosis  are  performed  on  animal  parts  (figure  37,  38). 
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• 

Gowning  and  gloving 

• 

Wound  Debridement 

• 

Handling  instruments 

• 

Tendon  repair 

• 

Tying  knots 

• 

Plastering  technique 

• 

Suturing 

• 

MAS 

• 

Mesenteric  dissection 

—  Troubleshooting 

• 

Anastomoses 

—  Instrumentation 

—  End-end 

—  Manipulation  tasks 

—  End-side 

—  Electrosurgery 

• 

Transverse  arteriotomy  repair 

• 

Safe  Surgery/  Teamwork/  Attitude 

• 

Longitudinal  arteriotomy  patch 

Figure  34.  Tasks  in  the  Royal  Australasian  College  of  Surgeons  Basic  Surgical  Skills  Course 


Participant  performance  is  measured  in  two  ways.  Assessment  was  performed  using  a  Likert 
scale,  with  5  being  no  errors,  4  occasional  errors  which  the  participant  corrected,  3  occasional  errors 
uncorrected,  2  frequent  errors  and  1  unable  to  proceed  without  instructions.  However  it  was  stressed 
that  the  inter-rater  reliability  of  this  system  had  yet  to  be  validated.  Nevertheless  the  level  of  supervision 
allowed  for  an  important  component  of  formative  assessment.  In  addition  to  this,  the  participants  self- 
assess  their  performance  in  pre-  and  post-course  measures  of  each  of  the  skills/tasks/procedures  using  a 
10  cm  visual  analogue  scale  (VAS). 


Figure  35  Knot  tying  Figure  36.  Abdominal  wound  closure 
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Figure  37  Excising  skin  lesion  Figure  38  End  to  side  anastomosis 

A  total  of  217  participants  in  8  venues  throughout  Australia  were  tested  with  a  tutor  to  participant  ratio 
of  1 :2  to  1 :3.  The  administration  of  the  training  and  evaluation  is  clearly  very  personnel  intensive.  Each 
of  the  tasks  was  assessed  for  the  overall  scores  comparing  the  pre-test  scores  and  post  test  scores.  In  the 
majority  of  the  test  there  were  high  levels  of  correlation  which  indicated  significant  improvement  during 
the  course. 
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Figure  39  Individual  correlations  of  the  skills  between  pre-test  and  post-test  evaluation 
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In  spite  of  the  validity  of  the  BSS  course,  there  are  a  number  of  issues  which  must  be  addressed. 
There  needs  to  be  a  common  language  to  share  assessment  evaluations.  The  tasks  which  are  used 
represent  a  pragmatic  approach,  and  more  scientific  exploration  is  needed.  The  courses  are  compulsory 
which  has  significant  resource  implication.  Distant  learning  has  raised  the  question  of  portability  vs 
centre-driven  courses.  There  must  be  curriculum  linkage  with  the  opportunity  to  practice  learned  skills. 

The  MISTELS  Program:  From  Conception  to  Credentialing  -  (Fried) 

For  the  training  of  young  surgeons,  operating  room  (OR)  time  is  an  expensive  and  limited  resource. 
Training  in  laparoscopic  surgery  involves  novel  skills.  Therefore  learning  opportunities  in  the  OR  can 
be  optimized  by  having  a  basic  level  of  skill  and  familiarity  with  instrumentation  and  skill  in  their  use. 
These  skills  would  best  be  acquired  in  a  relaxed  environment  outside  the  OR.  The  approach  taken  is  the 
McGill  Inanimate  Surgical  Training  and  Evaluation  for  Laparoscopic  Surgery  (MISTELS). 

The  acquisition  of  the  most  basic  laparoscopic  skills  requires  1.)  Familiarity  with  the  equipment  and 
2.)  Learning  basic  technical  skills  of  feel  for  tissue,  traction-countertraction,  ambidexterity,  cutting, 
suturing,  secure  hemostasis,  etc.  This  must  be  coupled  with  judgment  and  interpretation  of  anatomy  and 
operative  findings. 

Education  and  assessment  of  surgical  skills  have  evolved  over  time.  The  American  Board  of 
Surgery  required  intra-operative  assessment  of  surgical  skills  as  part  of  its  certification  process  until 
1952.  The  current  skills  assessment  requires  that  residents  maintain  an  operative  log  and  perform  a 
minimum  number  of  specific  surgical  procedures.  Final  assessment  of  surgical  skills  during  the 
residency  program  is  by  the  In  Training  Evaluation  Reports  (ITERs).  For  certification  in  surgery  the 
Royal  College  of  Surgeons  (Canada)  and  American  Board  of  Surgery  conduct  objective  examinations  to 
assess  clinical  knowledge  and  judgement.  There  is  no  objective  assessment  of  surgical  technical  skills. 
Scott,  et  al,  compared  scores  on  the  American  Board  of  Surgery  In  Training  Exam  (ABSITE)  to 
performance  in  a  skills  lab  and  to  intra-operative  assessment  of  surgical  skills.  The  conclusion  was  that 
there  was  no  correlation  between  ABSITE  scores  and  lab  or  intra-op  assessment  of  surgical  skills 
(Surgery  2000:  613-22).  This  raises  the  question  “Is  the  current  system  of  acquisition  and  evaluation  of 
surgical  skills  adequate?” 

There  has  been  a  recent  move  to  develop  objective  training  models  to  structure  the  acquisition  and 
assessment  of  surgical  skills  in  order  to  minimize  time  and  expense  while  maximizing  utilization  of 
resources.  One  answer  is  to  use  inanimate  training  methodologies,  which  would  provide  training  that  is 
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easy  to  schedule,  easily  reproduced,  provide  objective  measures  and  be  cost  effective.  This  need  lead  to 
the  development  of  the  McGill  Inanimate  System  for  Training  and  Evaluation  of  Laparoscopic  Skills 
(MISTELS)  Program.  The  program  is  based  upon  the  following  principles:  1).  Tasks  should  be 
derived  from  the  O.R.  skills,  2.)  The  skills  should  be  translated  to  operating  room,  3.)  Tasks  should 
utilize  actual  surgical  instruments  and  equipment,  4.)  Performance  must  be  objectively  measured,  5.) 
The  system  should  be  inexpensive,  portable  and  reproducible  and  6.)  Measurements  should  be  sensitive 
enough  to  distinguish  between  different  levels  of  competence. 

In  order  to  develop  a  reliable  program  it  was  necessary  to  develop  exercises  and  the  testing  process. 
From  these  objective  measuring  tools  were  derived,  which  were  subjected  to  reliability  and  validity 
studies.  For  the  development  of  exercises,  a  panel  of  “experts”  was  convened  to  review  videotapes  of 
laparoscopic  procedures  and  identify  components  of  frequently  performed  procedures  that  could  easily 
be  learned  and  tested  in  an  inanimate  environment.  In  addition  it  was  necessary  to  identify  skills  which 
were  associated  with  a  “learning  curve”.  Finally  it  was  important  to  use  optical  systems  identical  or 
“equivalent”  to  the  laparoscopic  system.  The  initial  program  began  with  an  introductory  video  tape  and 
7  standardized  exercises  performed  in  an  endoscopic  trainer  box  which  were  scored  for  precision  and 
speed. 


Figure  40  The  MISTEL  system  with  portable  and  desktop  images 

The  7  tasks  were:  1.)  Pegboard  transfer,  2.)  Pattern  cutting,  3.)  Application  of  hemostatic  clips, 
4.)  Placement  of  ligating  loops,  5.)  Placement  and  fixation  of  mesh  over  defect  6.)  Placement  of  simple 
suture  with  intra-corporeal  knot  and  7.)  Placement  of  a  simple  suture  with  an  extra-corporeal  knot 
The  pegboard  transfer  (figure  41)  consists  of  lifting  pegs  from  a  pegboard  with  one  hand, 
transferring  them  to  the  other  hand  and  placement  on  another  pegboard;  then  reversing  the  procedure. 
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The  purpose  is  eye-hand  coordination,  bimanual  dexterity  and  depth  perception.  Scoring  is  based  upon 
dropped  pegs  for  errors,  and  time  to  completion. 

The  pattern  cutting  (figure  42)  requires  the  student  to  cut  a  circular  pattern  out  of  a  4”  x  4”  gauze 
suspended  between  alligator  clips.  The  purpose  is  to  use  two  hands  to  apply  traction,  expose  best  angle 
to  cut,  and  to  cut  accurately.  Scoring  is  by  percent  area  cut  outside  of  pre-marked  line,  and  time  to 
completion  of  task 


Figure  41  Pegboard  transfer  Figure  42  Pattern  cutting 

The  application  of  hemoclips  (figure  43  a)  consists  of  placing  hemostatic  clips  on  a  tubular 
structure  at  pre-marked  positions,  and  cutting  between  the  clips.  The  purpose  is  to  replicate  a  similar 
procedure  in  surgery.  Clips  must  be  placed  securely  and  accurately.  Scoring  is  by  accuracy  of 
placement  of  clips,  with  penalty  for  insecure  clips,  measured  against  time. 

The  placement  of  ligating  loops  exercise  (figure  44b)  entails  placement  of  a  ligating  loop 
(endoloop)  at  a  marked  position.  The  purpose  is  to  replicate  placement  of  a  ligating  loop  during  surgery 
e.g.  appendectomy,  requiring  control  of  a  tubular  structure,  accurate  placement  and  secure  seating  of  the 
knot.  Scoring  is  accurate  position,  secure  knot  and  time  to  completion. 


Figure  43a  Application  of  hemoclip  Figure  43b  Application  of  ligating  loop 
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The  placement  and  fixation  of  mesh  over  a  defect  (figure  44)  requires  the  placement  of  a  mesh 
patch  over  a  defect.  The  purpose  is  to  replicate  placement  of  mesh  as  in  hernia  surgery,  including  secure 
fixation  with  accurately  placed  staples.  Scoring  is  the  percent  of  defect  covered,  security  of  fixation, 
number  of  staples  required  and  time  to  complete  task. 
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Figure  44.  Placement  and  fixation  of  mesh  over  defect 


The  placement  of  a  simple  suture  and  securing  it  with  an  intra-corporeal  and  extra-corporeal  knot 
(figure  45)  is  conducted  by  placing  a  simple  suture  through  marks  on  a  Penrose  drain  and  securing  of  the 
knot  with  intra-corporeal  or  extra-corporeal  technique.  The  purpose  is  to  develop  skill  at  introduction  of 
a  needle  into  a  trocar,  transfer  of  needle  between  needle  holders,  placement  of  suture,  and  knot  tying. 
Scoring  is  by  accuracy  of  position,  security  of  knot,  time  to  complete  knot. 


Figure  45  Placement  of  simple  suture  with  intra-cOrporeal  and  extra-corporeal  knot 

First  the  program  was  tested  with  inter-rater  reliability  (n=16:  ICC=0.991  with  95%  Cl:  0.973- 
0.997)  and  test-retest  reliability  (n=12:  ICC=  0.889  with  95%  Cl:  0.628-0.968).  Construct  validity  was 
performed  on  149  volunteer  academic  and  community  surgeons,  junior  to  chief  resident  and 


46 


.Metrics  Conference 


1  August  2, 2001 
Version  1.3 


laparoscopic  fellows  from  a  wide  geographic  distribution  (<  15  medical  schools  in  Canada  and  the  U.S.) 
with  a  wide  ethnic,  gender  and  handedness  diversity.  Junior  subjects  (n=77)  were  PGY  1,2,  and  3  and 
senior  subjects  were  PGY  4,5,  fellows  and  attending  surgeons.  The  results  clearly  show  improving 
scores  with  increased  experience  (figure  46)  to  prove  construct  validity. 
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Figure  46  Construct  validity  showing  increased  proficiency  with  years  of  training 


Concurrent  validity  evaluated  the  relationship  between  performance  in  an  inanimate  laparoscopic 
trainer  in  the  MISTELS  program  and  analogous  tasks  in  vivo  (figure  47).  The  experiment  consisted  of 
12  residents  who  were  evaluated  using  the  inanimate  laparoscopic  trainer  and  then  performed  analogous 
skills  in  a  live  pig  model.  Correlation  was  done  between  in  vitro  and  in  vivo  skills  to  determine  the 
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effect  of  practice  in  vitro  on  performance  in  vivo,  as  tested  in  6  of  the  12  residents.  All  tasks,  except 
the  loop  task,  showed  significant  correlation  to  the  p  =  0.05  or  better. 


Cutting  Clipping  Stapling  Mesh 


Intracorporeal  Knot  Extracorporeal  Knot 

Figure  47  The  in-vivo  tasks  corresponding  to  the  inanimate  tasks 
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Figure  48  Correlation  between  the  inanimate  and  in-vivo  tasks 

Content  validity  was  measured  against  subjective  evaluation  of  resident’s  performance  by  attending 
surgeons  during  a  clinical  rotation.  This  ITER  (In-training  Evaluation  Report)  is  a  consensus  of 
“expert”  surgeons  using  a  rating  scale  of  Unsatisfactory,  Borderline,  Average  or  Superior.  Fifty  surgical 
residents  who  had  their  ITER  in  the  same  academic  year  as  MISTELS  testing  had  4-6  independent 
evaluations  of  technical  skill  per  resident  per  year.  All  evaluations  were  “satisfactory”  or  “superior”. 
The  percent  “superior”  evaluations  versus  MISTELS  score  was  compared  (figure  49)  and  demonstrated 
that  the  low  and  high  scores  are  comparable. 

In  summary,  the  question  of  what  is  being  measured  appears  to  be  a  combination  of  innate  ability 
and  acquired  skills.  There  is  large  variability  at  first  testing,  with  decreased  variance  and  better 
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performance  with  experience.  Performance  could  be  measured  quantitatively.  Evidence  of  validity  is 
established  by  results  that  correlate  with  PGY  of  training,  with  seniors  performing  better  than  juniors 
and  correlates  to  performance  in  live  animals.  In  addition  the  scores  reflect  the  overall  evaluation  of 
surgical  skill  by  ITER.  A  longitudinal  follow-up  study  shows  each  resident  improves  in  MISTELS 
scores  as  he/she  progresses  through  residency  program.  Finally  the  MISTELS  program  has  a  high  inter¬ 
rater  and  test-retest  reliability 
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Figure  49  Content  validity  showing  MISTEL  SCORES  versus  low/high  expert  subjective  scores 
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Iterative  evaluations  resulted  in  a  few  of  the  tasks  being  discarded.  The  application  of  hemoclips  was 
not  sensitive  enough  (it  was  too  easy)  and  the  placement  and  fixation  of  mesh  over  a  defect  did  not  add 
substantially  to  validity  of  the  test  (by  multiple  regression  analysis)  and  was  very  expensive. 

Overall,  it  can  be  concluded  that  the  MISTELS  system  is  inexpensive,  easily  reproduced,  not 
labor  intensive,  portable,  reliable  and  has  been  repeatedly  validated. 

The  Objective  Structured  Assessment  of  Technical  Skill  (OSATS):  Five  years  of 
research  in  the  testing  of  technical  skills  in  surgery  (Regehr  &  Reznick) 

Assessing  procedural  competence  is  one  of  the  core  expectations  of  a  surgical  training 

program.  Traditionally,  this  assessment  has  been  enacted  through  a  preceptor’s  observation  of  a 
trainee’s  performance  in  the  operating  room.  However,  we  can  no  longer  rely  on  the  patient  context 
as  the  sole  venue  to  evaluate  (or  teach)  surgical  procedures.  There  are  many  reasons  for  this, 
including:  the  ethical  issues  in  using  patients  for  these  purposes,  the  high  cost  of  the  surgical  minute, 
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the  increasing  intolerance  of  error,  the  difficulty  in  standardizing  testing  conditions  and  the  difficulty 
in  ensuring  appropriate  coverage  of  relevant  procedures.  This  acknowledgement  has  led  to  a  variety 
of  efforts  around  the  world  to  develop  and  research  non-patient  platforms  for  teaching  and  testing  of 
procedural  skills  for  surgical  trainees. 

Many  of  these  efforts  have  involved  the  development  of  highly  sophisticated 
technology-based  simulators  such  as  computer  driven  mannequins  and  virtual  reality  modules. 
However,  at  the  University  of  Toronto  Department  of  Surgery  we  were  concerned  that  these  “high 
tech”  options  were  costly  (both  in  development  and  in  rollout),  making  it  difficult  for  many  schools 
to  afford  the  hardware  and  software  necessary  to  implement  these  solutions  on  a  large  scale.  Further, 
the  rapid  advances  in  the  technology  almost  guaranteed  that  by  the  time  a  model  was  ready  for 
rollout,  the  technology  on  which  it  was  based  would  be  out  of  date  rendering  it  sub-optimal  relative 
to  what  was  possible.  And  finally,  we  were  not  convinced  from  the  perspective  of  educational  theory 
that  such  highly  sophisticated  technological  solutions  were  necessary  for  the  development  and 
evaluation  of  basic  skills  and  procedural  knowledge  in  surgery.  Thus,  when  we  began  a  systematic 
program  of  research  into  the  teaching  and  testing  of  technical  skills,  the  program  of  research  was 
based  on  a  philosophy  of  exploring  the  limits  of  less  sophisticated,  less  expensive  approaches  to 
teaching  and  testing  technical  skills:  the  use  of  bench  model  simulations. 

For  the  purposes  of  developing  an  appropriate  evaluation  tool  to  assess  the  technical  skills  of 
surgical  residents,  we  adopted  the  format  of  the  Objective  Structured  Clinical  Examination  (OSCE) 
as  introduced  by  Harden.  The  OSCE  is  a  multi-station  examination  in  which  participants  are 
expected  to  perform  a  particular  clinical  task  in  a  relatively  short  time  period.  The  performance  of 
each  structured  task  is  observed  and  marked  either  by  a  content  expert  (such  as  a  clinical  faculty 
member)  or  by  an  individual  who  is  portraying  the  patient  with  whom  the  candidate  interacts. 

Starting  in  1994,  the  surgical  education  research  group  at  the  University  of  Toronto  began 
evaluating  the  feasibility,  reliability  and  validity  of  a  bench  model  version  of  the  OSCE  for 
evaluating  the  technical  skills  of  general  surgery  residents.  This  evaluation  tool  came  to  be  known  as 
the  Objective  Structured  Assessment  of  Technical  Skill  (OSATS).  There  have  been  several  previous 
reports  of  individual  studies  evaluating  the  OSATS.  This  report  compiles  and  summarizes  these 
findings. 

The  OSATS  is  a  multi-station  “bell-ringer”  examination  in  which  segments  of  6  to  8  surgical 
procedures  are  represented  at  a  series  of  “stations”  that  are  placed  in  a  circuit  around  a  room.  For 
each  procedure,  the  relevant  anatomical  structures  are  simulated  by  “bench  models”  that  are 
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constructed  from  materials  such  as  plastic,  metal  and  fabric  (Figure  50).  A  different  candidate  is 
present  at  each  station  and  each  is  given  instructions  regarding  the  particular  procedure  that  is  to  be 
performed  at  that  station.  All  candidates  simultaneously  perform  the  appropriate  segment  of  the 
procedure  in  a  15  minute  period.  Upon  completion  of  the  15  minute  period,  candidates  move  to  the 
next  station  and  the  process  is  repeated  until  all  candidates  have  completed  all  stations.  At  each 
station,  a  knowledgeable  operating  room  nurse  is  available  to  assist  the  candidate  and  a  qualified 
surgeon  observes  and  marks  the  candidates’  performance. 


Figure  50.  Sample  of  the  simulated  anatomical  structure  used  for  a  typical  OSATS  station 
(embolectomy)  built  from  foam,  fabric,  Penrose  drain  and  colored  liquid.  Only  the  Penrose  drain 
must  be  replaced  for  reuse  of  the  model. 

Examiners  use  two  marking  systems:  a  task-specific  binary  checklist  and  an  operation- 
independent  global  rating  scale.  The  task-specific  binary  checklist  analyzes  the  elements  of  surgical 
maneuvers,  enumerating  each  of  the  actions  that  are  deemed  relevant  to  the  segment  of  the  particular 
procedure  being  performed.  For  each  action,  the  examiner  indicates  whether  the  candidate  performed 
the  action  appropriately  or  not  and  the  score  on  the  checklist  is  the  total  proportion  of  items  identified 
as  having  been  performed  correctly  by  the  end  of  the  15-minute  period.  Because  the  task-specific 
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checklist  is  highly  idiosyncratic  to  the  particular  procedure,  a  different  checklist  is  created  and 
validated  by  a  group  of  content  experts  for  each  station.  The  global  rating  scale  taps  into  the 
constructs  of  surgical  performance,  identifying  several  dimensions  related  to  operative  performance 
such  as:  respect  for  tissue,  time  and  motion,  instrument  handling,  knowledge  of  instruments,  flow  of 
operation,  use  of  assistants,  and  knowledge  of  procedure.  Each  dimension  is  graded  on  a  5-point 
rating  scale  with  points  one,  three  and  five  anchored  by  explicit  behavioral  descriptors.  A  candidate’s 
score  on  the  station  is  the  average  of  the  marks  on  the  seven  dimensions  expressed  as  a  percentage  of 
the  total  possible  mark.  Because  the  items  on  the  global  rating  scale  are  intended  to  be  operation 
independent,  the  same  rating  scale  is  used  at  each  of  the  stations. 


Admin 

#  Progs 

#  Cands 

PGY 

#  Stns 

#  Exmrs 

1994 

1 

20 

1,3,5 

6 

12 

1995 

1 

48 

1-6 

8 

48 

1996 

2 

53 

2-5 

8 

54 

1997 

4 

65 

2-5 

8 

32 

1998a 

4 

36 

2-5 

8 

24 

1998b 

4 

41 

2-5 

8 

24 

Table  9.  Details  of  the  six  administrations  of  the  OSATS  between  1994  and  1998. 


•Excision  of  a  skin  lesion 

•Chest-tube  insertion 

•Vascular  anastomosis 

•Abdominal  wall  closure 

•Tracheostomy 

•Insertion  of  J-tube 

•Difficult  skin  closure 

•Pyloroplasty 

•Rectal  anastomosis 

•Control  of  IVC  hemorrhage 

•Hernia  repair 

•Ileostomy 

•Embolectomy 

•Insertion  of  a  T-tube 

•Choledocho-enterostomy 

•Hand  sewn  small  bowel 
anastomosis 

•Stapled  small  bowel 
anastomosis 

•Laparoscopic 

cholecystectomy 

Table  10  List  of  stations  used  over  the  six  administrations  of  the  general  surgery  OSATS  between 
1994  and  1998. 

Between  1994  and  1998,  the  OSATS  was  administered  6  times  to  a  total  of  263  general 
surgery  residents  from  12  programs  across  North  America.  Candidate  training  levels  ranged  from 
post-graduate  year  1  (PGY1)  to  PGY6,  with  the  majority  of  administrations  including  residents  from 
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PGY2  to  PGY5.  A  list  of  the  administrations  is  presented  in  Table  9.  A  list  of  the  stations  developed 
and  used  in  the  context  of  these  administrations  is  presented  in  Table  10. 

Reliability  refers  to  the  precision  of  the  scores  generated  by  the  examination.  It  functions  as  a 
form  of  signal-to-noise  ratio  where  the  signal  is  the  individual’s  “true  ability”  and  the  noise  is  the 
error  in  measuring  that  true  ability.  A  reliability  of  zero  indicates  that  scores  generated  by  the 
examination  are  simply  noise  (random  error)  and  a  reliability  of  one  indicates  that  the  scores 
generated  by  the  examination  is  a  complete  signal  (based  on  true  ability).  Reliability  can  be  evaluated 
in  several  ways.  It  can,  for  example,  assess  the  level  of  precision  between  two  independent  evaluators 
(inter-rater  reliability).  It  can  also  assess  the  precision  of  measurement  across  many  stations,  inferred 
by  the  extent  to  which  excellent  candidates  perform  well  across  many  similar  tasks  and  poor 
candidates  perform  poorly  across  many  similar  tasks  (internal  consistency).  Two  forms  of  reliability 
have  been  assessed  for  the  OSATS. 

For  each  administration  of  the  examination,  the  internal  consistency  of  the  examination  across 
stations  was  calculated  using  Cronbach’s  alpha  coefficient.  The  average  inter- station  alpha  across  the 
six  iterations  for  the  checklist  scoring  method  was  0.65  (ranging  from  0.33  to  0.79)  and  for  the  global 
rating  method  was  0.82  (ranging  from  0.75  to  0.89). 

In  the  1994  and  1996  administrations,  two  examiners  were  placed  at  each  station  in  order  to 
assess  the  inter-rater  reliability  of  the  examination.  To  assess  inter-rater  reliability,  the  two  examiners 
in  each  station  were  asked  to  mark  the  candidate  independently.  Each  pair  of  examiners  was 
specifically  asked  to  avoid  interacting  with  each  other  for  the  duration  of  the  examination  day,  both 
during  an  examinee’s  performance  (to  avoid  cross-examiner  contamination  of  individual 
assessments)  and  between  examinees  (to  avoid  convergent  drift  in  the  use  of  the  scales  over  the 
course  of  the  examination).  For  each  station,  scores  from  the  two  examiners  were  compared  using  an 
intra-class  correlation  coefficient  (ICC)  with  the  examiner  as  a  random  factor  in  the  model.  In  the 
1994  administration  the  average  inter-rater  ICC  across  the  six  stations  was  0.65  for  the  checklist 
scores  and  0.70  for  the  global  rating  scores.  For  the  1996  administration,  only  a  global  rating  was 
used  by  one  of  the  examiners  and  the  average  inter-rater  ICC  across  the  eight  stations  was  again  0.70 
for  the  global  rating  scale. 

Validity  refers  to  the  extent  to  which  the  test  is  measuring  what  one  thinks  it  is  measuring. 
Several  methods  have  been  used  to  establish  the  validity  of  the  OSATS.  First,  for  each  iteration  of 
the  examination  it  was  possible  to  assess  the  construct  validity  by  comparing  the  scores  of  candidates 
at  various  levels  of  training  experience.  Table  1 1  presents  the  mean  checklist  and  global  scores  by 
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year  of  training  across  the  six  administrations.  For  each  administration,  the  simple  linear  Pearson’s 
correlation  was  calculated  between  PGY  level  and  the  checklist  and  global  rating  scores.  Across  the 
six  administrations  the  mean  correlation  of  PGY  status  with  checklist  scores  was  0.65  (ranging  from 
0.38  to  0.76)  and  with  global  rating  scores  was  0.70  (ranging  from  0.46  to  0.84).  Thus,  training  level 
is  accounting  for  approximately  40%  to  50%  of  the  variance  in  checklist  and  global  rating  scores, 
suggesting  quite  reasonable  construct  validity  for  both  measures. 


Table  11  OSATS  scores  as  a  function  of  year  of  post-graduate  training  collapsed  across  the  six 
administrations  of  the  examination  between  1994  and  1998. 

In  addition  to  construct  validity,  several  measures  have  been  used  to  establish  the  concurrent 
validity  of  the  examination  scores.  In  1994,  the  identical  set  of  six  stations  was  administered  in  both  a 
bench  model  format  and  in  a  live  animal  format  (using  pigs).  The  Pearson’s  correlation  between  the 
bench  and  live  versions  of  the  examination  (when  disattenuated  for  the  unreliability  of  the 
examination  scores  themselves)  was  0.69  for  the  checklist  scores  and  was  0.71  for  the  global  rating 
scales,  suggesting  reasonable  comparability  between  the  bench  model  and  live  model  versions  of  the 
examination. 

In  1996,  an  additional  measure  of  operative  quality  was  created:  an  analysis  of  the  final 
product  generated  by  candidates  at  each  station.  For  six  of  the  eight  stations,  the  final  product  of  the 
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resident  performance  was  collected  and  removed  to  a  separate  room.  The  final  products  for  each 
station  were  evaluated  by  a  pair  of  surgeons  who  were  blinded  to  the  resident’s  year  of  training  and  to 
the  checklist  and  global  scores  obtained  by  the  resident  during  the  performance  of  the  procedure.  The 
quality  of  each  final  product  was  evaluated  using  four  five-point  rating  scales  that  measured 
completeness,  aesthetics,  anticipated  functionality  and  overall  quality.  Upon  completion  of  the 
examination,  each  resident  received  a  final  product  score  which  was  the  average  of  the  six  scores  s/he 
had  received  for  the  six  individual  products  evaluated.  The  Pearson’s  correlation  of  final  product 
scores  with  the  scores  on  the  OSATS  was  0.63  for  the  checklist  scores  and  0.74  for  the  global  rating 
scores.  This  quite  reasonable  set  of  correlations  provided  evidence  of  concurrent  validity  both  for  the 
OSATS  and  for  the  final  product  measure,  which  itself  has  since  been  examined  as  a  potential 
independent  evaluation  tool. 

The  first  two  measures  of  concurrent  validity  gave  some  credibility  to  the  examination  scores 
generated  in  the  OSATS.  In  each  case,  however,  the  concurrent  measures  were  associated  with  an 
extra-operative  evaluation  format.  We  were  also  interested  in  determining  the  extent  to  which  scores 
on  the  OSATS  related  to  performance  in  the  clinical  setting.  However,  no  obvious  measures  of 
procedural  competence  in  the  clinical  setting  were  available.  The  traditional  method  of  evaluating 
procedural  competence  has  been  the  in-training  evaluation  report  (ITER),  whereby  clinical  faculty 
supervisors,  at  the  end  of  a  clinical  rotation,  give  a  generalized  impression  of  a  resident’s 
performance  in  a  number  of  domains.  While  the  ITER  is  based  on  multiple  observations  over  an 
extended  period  of  interaction,  these  scores  are  generally  thought  to  have  low  reliability,  limiting 
their  value  as  an  assessment  tool  and,  for  our  purposes,  a  concurrent  validity  measure.  In  response  to 
this  concern  a  forced  choice  ranking  procedure  was  developed  in  an  effort  to  obtain  a  more  useful 
measure  of  faculty  opinion  of  residents’  technical  abilities.  This  procedure  asked  clinical  supervisors 
to  compare  the  operative  skills  of  two  residents  directly  rather  than  rating  each  resident 
independently.  Through  multiple  paired  comparisons  across  many  pairs  of  residents  and  many  faculty 
members,  a  rank  order  of  residents  was  created.  This  rank  order  was  structured  to  maximize  the 
consistency  of  faculty  opinion  at  the  pairwise  level.  In  1996,  a  forced  choice  ranking  of  18  PGY4  and 
PGY5  residents  was  created  and  compared  to  the  ranking  of  those  individuals  on  the  OSATS.  The 
Spearman  rank  order  correlation  between  the  clinical  ranking  and  the  OSATS  ranking  based  on  the 
checklist  ratings  was  0.72  and  with  the  OSATS  ranking  based  on  the  global  ratings  was  0.89, 
suggesting  high  concurrent  validity  for  the  checklist  scores,  the  global  scores  and  the  forced  choice 
rankings. 
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As  described  here,  the  OSATS  was  successfully  administered  as  a  general  surgery 
examination  for  four  consecutive  years  between  1994  and  1997.  In  the  fourth  year,  the  exam 
evaluated  65  residents  from  four  different  schools  across  Ontario  and  used  examiners  from  all  these 
schools  with  no  change  in  the  psychometric  properties  of  the  examination.  In  1998  the  examination 
was  exported  to  two  sites  in  the  United  States.  For  each  of  those  administrations,  the  University  of 
Toronto  provided  technical  support,  but  the  examination  was  largely  coordinated  at  the  local  site.  At 
both  US  sites,  the  administration  was  jointly  supported  by  four  separate  institutions,  which  supplied 
both  resident  candidates  and  surgical  faculty  examiners  for  the  examination.  These  two  extramural 
experiences  have  strengthened  our  belief  about  the  generalizability  of  our  examination  results  and 
have  confirmed  the  feasibility  of  a  model  of  central  examination  preparation  and  peripheral  delivery. 
In  addition,  the  format  has  since  been  effectively  adapted  for  obstetrics  and  gynecology  in  North 
America.  Thus,  its  feasibility  as  an  evaluation  tool  appears  well  established. 

Further,  the  OSATS  is  now  being  used  frequently  at  our  own  institution  as  an  outcome 
measure  for  evaluating  the  effectiveness  of  teaching  innovations,  and  has  been  adopted  by 
researchers  in  the  United  Kingdom  as  a  gold  standard  for  evaluating  new  measurement  instruments 
such  as  the  motion  analysis  measures  of  the  Imperial  College  Surgical  Assessment  Device  (ICSAD). 
Adaptations  of  the  OSATS  are  being  used  for  evaluating  surgeons  who  have  been  identified  as 
having  possible  problems  in  the  technical  domain  and  is  being  explored  as  a  testing  platform  for 
learning  and  performing  new  procedures  in  the  context  of  clinical  trials.  Thus,  it  appears  that  the 
applicability  of  the  OSATS  is  becoming  increasingly  widespread. 

In  summary,  based  on  5  years  of  research,  our  data  indicate  that  we  have  been  able  to  apply 
the  OSCE  format  to  the  domain  of  surgical  skills.  We  have  developed  a  non-patient,  lab-based 
platform  for  testing  technical  skills  that  demonstrates  good  reliability,  established  construct  and 
concurrent  validity,  widespread  feasibility,  and  demonstrable  applicability.  Further,  we  have 
effectively  established  this  examination  in  the  context  of  a  bench  model  platform.  The  models  we 
have  used  have  varied  in  sophistication  from  very  simplistic  and  inexpensive  representations,  to  fairly 
realistic  reproductions  with  reasonable  anatomic  fidelity.  Data  from  our  research  group  on  simulation 
fidelity  seem  to  be  indicating  that,  for  early  surgical  learners,  low  fidelity  simulators  work  as  well  as 
do  high  fidelity  simulators  in  teaching  basic  surgical  manipulations.  As  of  yet,  we  have  not  explored 
the  utility  of  very  costly  approaches  such  as  virtual  reality  simulations. 

Rosser  Minimally  Invasive  Skills  and  Suturing  Program  (Rosser) 
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The  Rosser  Minimally  Invasive  Skills  and  Suturing  Program  targets  endoscopic  suturing  as  the 
critical  outcomes  measure  because  it  is  the  most  difficult  task  to  perform  in  the  laparoscopic 
environment  and  requires  the  possession  of  advanced  skill  set  to  perform  the  task  on  a  superior  level. 
However,  once  endoscopic  suturing  is  mastered,  the  surgeon  has  accomplished  a  tremendous  advance 
when  performing  advanced  minimally  invasive  procedures.  Laparoscopic  suturing  requires  accurate 
targeting,  2-D  depth  perception  compensation,  bimanual  dexterity  and  two-handed  choreography. 
Traditionally  laparoscopic  suturing  is  thought  to  require  intensive  and  prolonged  training,  but  this  total 
program  has  been  devised  to  train  laparoscopic  surgery  in  a  short  time.  The  program  is  devised  to  be 
very  efficient  (1.5or  2.5  day  program)  but  it  must  be  supervised  (student/instructor  ratio  is  4  to  1). 
Preparatory  skill  development  has  emphasis  on  non-dominant  hand  skill  development.  The  program 
begins  with  didactic  lectures  and  a  compact  disc-  read  only  memory  (CD-ROM)  assisted  curriculum 
with  detailed  deconstruction  of  the  suturing  process.  This  instruction  is  followed  by  the  technical  skills 
portion  and  finishes  with  an  animal  laboratory.  To  date  a  large  data  base  with  metric  follow-up  is 
currently  available  online  to  provide  post  course  development  evaluation.  In  addition  a  distant 
education  program  is  available.  After  extensive  evaluation,  three  primary  technical  skill  drills  have  been 
determined  to  be  effective  teaching  tools:  The  “cobra  rope”,  “pea  drop”  and  “triangle  transfer”.  The 
cobra  rope  (figure  50)  consists  of  two  handed  choreography,  targeting  and  non-dominant  hand 
development  to  instill  proper  suture  directional  manipulation.  The  pea  drop  (figure  51)  requires  2-D 
depth  perception  compensation,  non-dominant  hand  development  and  targeting  (increasing  accuracy 
when  grasping  tip  of  needle  during  suturing  process).  The  triangle  transfer  (figure  52)  develops  non¬ 
dominant  hand  skills,  2-D  depth  perception,  targeting  to  establish  mechanics  for  atraumatic  needle 
passage  through  tissue  during  the  penetration  phase  of  suturing 


Figure  50  Cobra  Rope  Drill  Figure  5 1  Pea  Drop  Drill  Figure  52  Triangle  Transfer  Drill 
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To  demonstrate  validitiy,  150  board  eligible  or  board  certified  surgeon  with  extensive  laparoscopic 
cholecystectomy  experience  were  tested  Face  validity  was  proven  because  the  individual  components 
of  suturing  were  deconstructed  in  order  to  design  the  drills.  Initially  (T=0),  82%  took  longer  than  five 
minutes  or  could  not  complete  an  intracorporeal  knot.  After  completion  of  the  didactic  and  drills,  it  was 
demonstrated  that  preparatory  drills  had  a  high  correlation  with  performance  of  suturing.  The  learning 
curve  of  both  the  three  drills  and  the  knot  tying  were  comparable  (figure  53).  This  demonstrated  that 
outstanding  suturing  abilities  can  be  achieved  in  a  short  period  of  time  with  this  program. 


differences  between  attempts  1  ana  2 ana attempts  2  and  IT  were 
significant  (?<.0Q1  for  botft.  unpaired  Student  i  test).  Values  are  given  as 
tne  mean-SEM. 


Figure  53  The  learning  curve  of  the  cobra  rope  drill  and  the  intra-corporeal  knot  tying 


For  construct  validity,  dexterity  drills  and  suturing  were  compared  and  demonstrated  a 
significant  difference  between  trained  surgeons  and  pre-residency  students  (figure  54).  The  correlation 
coefficient  between  the  suturing  tasks  and  each  of  the  drills  was  r=  0.051  and  0.053  with  p<0.001. 
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Table  1.  Dexterity  Drills  and  Suturing  Exercises,  Trained  Surgeons  vs  Residents* 


Resident  Subgroups! 


Dexterity  Drills  and 
Suturing  Exercises 

Trained  Surgeons 
(n  =  291) 

All  Residents 
(ns  99) 

General  Surgery 
Residents 
(n  =  6i) 

Gynecology 
Residents 
(it  *19) 

Preresidency 
Participants 
(n  *  19) 

Time 

i 

P 

1 

Time 

i 

P 

Time 

P  1 

Time 

i 

P 

Rope  pass  drill 

690  ±10 

676  ±18 

<.001 

687  ±23 

<.001 

630  ±44 

<.001 

681  ±42 

<.001 

Triangle  transfer  drill 

445  ±9 

385  ±16 

<.005 

389  ±20 

<.05 

346  ±37 

<.05 

411  ±38 

<001 

Old  cup  drop  drill! 

890  ±23  (n  ^  145) 

1005  ±43  (n  =  43) 

<.05 

...§ 

New  cup  drop  drill! 

1233  ±30  (n- 146) 

1371  ±48  (n»  56) 

<.05 

♦ . . 

Suturing  exercises 

1839  ±38 

1806  ±67 

<.001 

1673  ±  84 

<.001 

1555  ±156 

<.001 

2466  ±19 

<.001 

*  Time  in  seconds,  mean  ±  SEM,  required  to  complete  10  drills  and  suturing  exercises  by  trained  surgeons,  residents .  and  subgroups  of  residents. 
t  Completion  times  of  residents  as  a  group  and  of  3  subgroups  of  residents  are  compared  with  those  of  trained  surgeons  by  P  values  derived  with  unpaired 
2’tailed\  tests. 

tin  the  old  cup  drop  drill  and  the  new  cup  drop  drill  the  numbers  of  participants  are  listed  separately  from  the  numbers  in  other  types  of  drills  and  exercises, 
§ Ellipses  indicate  category  of  data  not  applicable. 


Figure  54  Comparison  of  trained  surgeons  and  students  on  dexterity  drills  and  laparoscopic  suturing 


The  effectiveness  of  CD-ROM  multimedia  tutorial  in  transferring  cognitive  knowledge  for 
laparoscopic  skill  training  was  evaluated.  (American  Journal  ofSurg.  Vol  1 79(4),  April  2000).  Two 
hundred  fifty  one  trained  surgeons  in  four  groups  (US  trained,  US  residents,  Greek  surgeons,  US 
surgeons  with  didactic  lectures)  were  given  either  the  didactic  lectures  or  the  CD-ROM  tutorial  (with  the 
same  author  for  both  methods  of  teaching)  in  order  to  compare  the  transfer  of  cognitive  skills  between 
the  two  methods.  The  results  demonstrated  that  the  CD-ROM  is  as  effective  as  traditional  didactic 
lecturing.  The  Greek  surgeons  showed  depressed  cognitive  transfer  but  excellent  skill  transfer,  perhaps 
due  to  the  language  differences.  This  observation  implies  that  future  tutorials  need  to  be  in  the 
participants’  first  language;  it  also  it  indicates  that  a  distant  education  curriculum  may  be  possible. 
Therefore,  a  complete  distant  learning  package  (figure  55),  based  upon  the  CD-ROM  multimedia 
approach  validated  above,  along  with  the  standard  “box”  trainer  for  the  Rosser  drills,  has  been  created 
and  is  being  implemented. 
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Figure  55  Distant  learning  package  Figure  56  The  Gabriel-Rosser  Inanimate  Proctor  (GRIP) 

In  order  to  include  objective  assessment  of  technical  skills  with  automatic  error  detection,  the 
Gabriel-Rosser  Inanimate  Proctor  (GRIP)  has  been  developed  (figure  56).  Whenever  the  student  makes 
moves  which  are  too  uncontrolled,  the  instrument  contacts  the  side  of  the  platform  and  registers  an  error. 
Similar  modifications  to  the  cup  drop  (figure  57)  and  triangle  transfer  (figure  58)  have  been  made. 


Figure  57  Automatic  error  detection  on  cup  drop  Figure  58  Automatic  error  detection  on  triangle  transfer. 

A  competitive  program  that  serves  as  a  focal  point  places  skill  development  in  the  spotlight  during 
conferences  and  meetings.  Resident  and  trained  surgeons  alike  are  challenged  to  showcase  their  skills. 
This  is  a  high  profile  recruitment  opportunity  to  encourage  the  implementation  of  standardized  skill 
development  curriculums.  Since  1996  over  800  surgeons  have  participated,  including  being  featured  at 
the  annual  meetings  of  the  American  College  of  Surgeons  (ACS),  the  Society  of  American 
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Gastrointestinal  Surgeons  (SAGES)  and  the  Society  of  Laparoscopic  Surgeons  (SLS).  This  competition 
has  been  designated  the  “Top  Gun”  Laparoscopic  Skills  Shoot  Out  (figure  59),  providing  both  an 
opportunity  of  skills  assessment  with  entertainment  (edutainment). 


Figure  59  The  Top  Gun  competition  and  awards  at  the  annual  ACS  meeting 

In  summary,  the  Rosser  program  is  a  comprehensive  approach  that  is  designed  to  establish  the 
advanced  skill  of  intracorporeal  suturing  and  anastomosis  in  a  very  short  period  of  time.  It  is 
applicable  to  novices  and  experienced  surgeons  alike.  It  has  a  standardized  easily  exportable  format 
with  distant  education  capabilities.  It  has  a  large  Internet  accessible  data  base  for  initial  and  follow- 
evaluation  of  progress. 

Trauma  Skills  Training:  Teaching  ATLS  with  Simulators  (Kaufmann) 

The  United  States  military  has  established  the  National  Capital  Area  Medical  Simulation  Center 
(NCAMSC)  of  the  Uniformed  Services  University  of  the  Health  Sciences  in  Bethesda,  MD.  This  Center 
is  one  of  the  most  comprehensive  learning  centers,  which  includes  the  three  components  of  medical 
education  skills  (cognitive,  personal/communication  and  technical)  into  a  single  institution.  There  is  a 
computer  laboratory  area  (for  testing  of  cognitive  skills  in  standard  pedagogical  format),  a  clinical 
simulation  area  (examination  rooms  with  video  cameras  for  observation  of  clinical  communication  skills 
while  examining  patient  actors)  for  Objective  Structured  Clinical  Exam  (OCSE)  of  “standardized 
patient”  exercises  and  a  surgical/virtual  reality  simulation  area  (with  simulators  for  training  and 
evaluation  of  technical  skills). 
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For  learning  technical  skills,  there  is  the  dedicated  surgical  simulation  laboratory,  which  includes 
the  following  simulators:  Telepresence  Surgery  System  (SRI,  International),  Anastomosis  Simulator 
(Boston  Dynamics,  Inc),  Limb  Trauma  Simulator  (MuscluoGraphics,  Inc),  Surgical  Workbench  (Hand 
Immersive,  Inc),  CathSim  &  PreOp  Simulators  (Immersion  Medical,  Inc),  UltraSim  Ultrasound 
Simulator  (MedSim,  Inc),  Patient  Anesthesia  Mannequins  (Laerda,  Inc.,  and  MedSim  Inc)  (figure  60) 
and  LapSim  (Surgical  Science,  Inc)  (figure  61).  In  addition,  intra-mural  research  is  improving  upon  the 
systems  as  well  as  developing  new  prototype  simulators. 

The  NCAMSC  is  the  only  institution  authorized  to  teach  the  Advanced  Trauma  and  Life  Support 
(ATLS)  course  by  substituting  simulators  (both  mannequin  and  VR  technology)  for  cadavers  or 
anesthetized  animals  in  the  technical  skills  practicum.  This  program  is  an  experimental  effort  that 
attempts  to  decrease  the  number  of  animals  that  will  need  to  be  used  for  training  of  surgical  skills.  The 
goal  of  the  ATLS  course  is  to  create  a  standardized  protocol  for  the  first  hour  of  trauma  patient  care. 
This  course  serves  as  a  common  language  for  physicians  from  different  countries  working  together. 
Internationally,  approximately  25,000  physicians  from  35  countries  are  trained  and  evaluated  each  year. 
Thus,  the  ATLS  serves  as  an  excellent  first  proof-of-concept  for  the  evaluation  of  initiating  simulation 
into  the  surgical  educational  process. 


Figure  60  Patient  Anesthesia  Mannequin  Figure  61  Virtual  Reality  simulator  (LapSim) 


Currently,  the  4  technical  skills  in  the  practicum  (diagnostic  peritoneal  lavage  (DPL), 
pericardiocentesis,  tube  thoracostomy  and  surgical  airway)  are  upon  animals  or  cadavers.  The 
NCAMSC  has  developed  two  simulators  -  the  DPL  (figure  62)  and  pericardiocentesis  (figure  63). 
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Figure  62  Diagnostic  Peritoneal  Lavage 


Figure  63  Pericardiocentesis 


The  Center  Integration  of  Medicine  and  Innovative  Therapy  (CIMIT™ )  is  a  non-profit  consortium  of 
world-leading  academic  and  research  institutions  founded  by  Partners  Healthcare  System,  Massachusetts 
General  Hospital  (MGH),  Brigham  and  Women’s  Hospital,  Massachusetts  Institute  of  Technology,  and 
Draper  Laboratory).  CIMIT  is  partnering  with  NCAMSC  in  developing  a  chest  tube  simulator  and  a 
surgical  airway  simulator  is  under  development  in  collaboration  with  the  Cleveland  Clinic  Research 
Center.  In  a  preliminary  assessment,  9  participants  (6  surgery  residents  and  3  surgical  staff)  who  had 
previously  taken  ATLS,  participated  in  the  ATLS  course  using  the  simulators  instead  of  animals  for  the 
skills  practicum.  The  survey  (Likert  scale)  revealed  that  7  believed  simulation  was  as  good  or  better  (3 
better,  4  as  good)  and  two  thought  the  simulators  to  be  not  as  good.  Further  studies  are  designed  to 
evaluate  the  validity.  There  is  excellent  face  validity,  since  the  images  (see  above)  very  closely  resemble 
the  patient  encounter.  These  simulators  teach  and  test  the  identification  of  correct  site,  angle,  depth,  and 
sequence  for  needle  insertion  tasks  and  can  be  generalized  to  other  (Seldinger)  needle-based  procedures. 
While  a  bit  more  complex,  the  chest  tube  and  surgical  airway  simulators  will  be  able  to  be  incorporated 
into  the  ATLS  format. 


Endoscopic  Sinus  Surgery  Simulator  (ES3)  (Sinanan) 

The  Endoscopic  Sinus  Surgery  Simulator  (ES3)  is  a  complete  surgical  simulator  (figure  64)  for 
otolaryngological  sinusoscopy  surgery.  It  is  comprised  of:  1.)  An  Immersion  6  degree  of  freedom  (DOF) 
sinusscope  (external)  and  operative  instrument  (internal,  within  a  mannequin  head)  (figure  65) ,  2.)  3-D 
segmented  surface  models  from  Visible  Human  Dataset  with  surface  textures  for  left  and  right  nasal 
sinuses  (figures  66, 67,  68) ,  3.)  Haptic  system  control  and  4.)  Voice  audio  feedback  and  simulated 
heartbeat. 
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Figure  64  ES3  simulator  evaluation  Figure  65.  ES3  mannequin  Figure  66  Abstract  tasks 


Figure  67  Sinus  anatomy  with  overlay  of  guides  Figure  68  High  fidelity  of  sinus  anatomy 


The  curriculum  is  comprised  of  fundamental  skills  which  are  presented  as  the  subtasks  of  navigation, 
injection  and  dissection.  Metrics  are:  1.)  Time  to  subtask  and  overall  task  completion  time,  2.) 

Accuracy  (computed  score),  3.)  Errors  and  4.)  Position  tracking  from  frame  to  frame.  There  are  three 
levels  -  novice,  intermediate  and  advanced.  In  the  novice  and  intermediate  modes,  there  are  training  aids 
of  3-D  graphic  overlays  (circular  hoops  for  desired  endoscopic  trajectory,  bulls  eye  targets  for  injection 
sites  and  text  labels).  The  training  is  taken  through  the  levels  of  increasing  difficulty.  The  novice  level 
(abstract  geometry  +  training  aids)  (figure  66)  is  designed  for  task  practice,  tool  familiarization  (needles, 
various  cutting  forceps,  sickle  knife,  suction,  microdebrider),  and  eye-hand  coordination  (geometry, 
hoops,  bulls  eye  targets).  The  intermediate  level  (anatomically  accurate  +  training  aids)  (figure  67) 


trains  and  assesses  guided  navigation  (e.g.  optimal  trajectory),  anatomy  and  guided  injection.  In  the 
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advanced  level  (anatomically  accurate)  the  focus  is  on  integration  of  skills  into  procedures.  There  is 
haptic  and  audio  feedback  but  no  guides  and  the  anatomy  has  augmented  tasks,  such  as  polyps.  By 
having  increasing  difficulty  and  training  aids,  it  is  possible  to  “shape  behavior”  to  perform  tasks 
correctly.  Evaluation  was  performed  on  three  groups:  1 .)  Nonphysicians  -  12  engineers  (novice  and 
intermediate  models  only),  2.)  Non-ENT  physicians  -  8  (novice  to  advanced)  and  3.)  ENT  surgeons 
(novice  to  advanced)  comprised  of  4  staff  ENT  surgeons  and  8  resident  trainees.  The  metrics  were 
efficiency  (time-to-subtask  and  overall  task  completion),  trajectory  path  analysis,  accuracy  (measure  of 
fidelity  to  assigned  goals  of  procedure  or  task  to  meet  an  established  criterion)  and  error  (negative 
measure,  proportional  to  clinical  significance).  The  scores  included  overall  score  (composite  value); 
accuracy  (of  eye-hand  coordination;  visio-spatial  integration  and  anatomic  knowledge  (advanced 
model));  hazard  score  (for  anatomic  knowledge  and  visio-spatial  integration);  sphere  path  score  (for 
efficiency  of  movement,  eye-hand  coordination  (novice  and  intermediate)  and  integration  of  task  to 
anatomy  (advanced).  Face  and  construct  validity  were  demonstrated  (figure  69)  with  correlation  with 
experience  of  the  residents  and  faculty. 
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Figure  69.  Face  and  construct  validity  with  correlation  values 

Motion  analysis  was  also  performed.  It  is  clear  by  the  trajectories  (figure  70)  that  complete 
random  motion  of  the  novice  consolidated  to  a  tighter  knot,  followed  by  very  deliberate  motions  in 
specific  areas  by  the  experienced  surgeon. 
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ESS  Simulation 


Figure  70  Motion  trajectories  of  novice,  trainee  and  experienced  surgeons. 


The  advanced  level  provides  a  training  and  evaluation  system  for  senior  residents  and  surgeons. 
There  is  a  direct  effect  of  receiving  training  on  the  simulator  with  the  ratings  of  participants  (figure  71). 
Those  with  no  simulator  training  performed  worse  than  those  with  high  ratings  on  the  simulator,  with 
the  experienced  surgeons  performing  best  of  all.  This  indicates  that  the  ES3  simulator  has  both  content 
and  concurrent  validity.  There  needs  to  be  validation  of  predictive  validity,  by  comparing  simulator 
performance  to  OR  performance. 
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Figure  71  Advanced  Level  -  effect  of  training  on  simulation. 


The  conclusion  is  that  the  ES3  simulator  provides  a  valuable  training  and  evaluation  tool.  The 
metrics  logically  derive  from  the  nature  of  the  simulator,  which  incorporates  measures  of  efficiency, 
accuracy,  and  error.  There  is  evidence  of  face,  construct  and  content  validity  in  that  performances 
correlates  with  prior  experience,  training,  handedness,  exposure  to  similar  task.  The  simulator  has  an 
embedded  training  curriculum  to  standardize  the  baseline  knowledge.  The  initial  orientation  from 
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experts  in  the  procedure  was  refined  by  subject  survey  and  performance  assessment.  It  remains 
necessary  to  validate  objective  measure  of  clinical  performance. 

The  Holy  Grail  of  Surgical  Simulation:  Teaching  &  Assessing  Technical  Judgment  (Heinrichs) 

A  different  approach  to  training  and  evaluation  of  surgical  skills  harkens  back  to  ancient  times. 

In  500  BC,  the  Hindu  physician  Sushruta.  (Veith  &  Zimmerman,  1990)  defined  the  eight  (a  ninth  is 
added  in  modem  surgery)  basic  tissue  manipulations  of  all  of  surgery.  They  are: 

•  Probing  -  Exploration  of  a  stmcture  by  visualization  and/or  palpation,  including  blunt  dissection 

•  Aspiration/Injection  -  Penetration  into  a  stmcture  for  removal  or  introduction  of  fluid 

•  Incision  -  Opening  into  structures  &  spaces  by  cutting 

•  Evacuation  -  Emptying  of  an  anatomic  space,  or  cavity 

•  Scaraification  -  Purposeful  injury  of  tissues 

•  Extraction  -  Removal  of  a  part  from  its  anatomic  site 

•  Excision  -  Complete  removal  of  a  stmcture  by  cutting 

•  Closure  -  Binding  together,  or  apposition  of  surfaces 

•  Implant-  /Transplantation  -  Introduction  of  a  medical  device  or  organ 

Interactions  of  instmments  with  tissues  are  the  fundamental  deformations  to  be  modeled  in  physics- 
based  simulators.  Instmction  frames  provide  structured  information  in  a  computer  instmction  device  that 
defines  specific  manipulations,  each  with  a  specific  instmments  and  descriptions  of  their  actions.  Each 
tissue  manipulation  has  various  approaches  using  different  surgical  tools.  Four  levels  of  evaluation  of 
manipulations  are  proposed  -  approach  to  manipulation,  execution  of  manipulation,  consequence  of  the 
manipulation,  and  exercise  of  safety  precaution  during  the  manipulation.  Within  each  of  these  levels, 
three  or  more  queries  are  proposed  to  define  the  metrics  for  training  and  evaluation. 

Level  ONE:  Approach  to  manipulations.  Examples  of  queries  are: 

Was  an  appropriate  instrument  selected  for  the  incision  task? 

Was  the  appropriate  site  chosen  to  make  the  incision? 

Was  the  site  stabilized  appropriately  before  making  the  incision? 

Level  TWO:  Execution  of  manipulations 

Was  an  appropriate  angle  of  approach  used? 

Was  the  length  of  the  incision  appropriate  for  the  immediate  need 

Did  the  initial  instmctions  meet  the  exposure  requirements  for  the  entire  procedure? 
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Level  THREE:  Consequence  of  interventions 

Did  the  instructions  of  actions  contribute  to  overall  efficiency? 

(  1 -sub-optimaL,  2-  marginal,  3-optimal,  4-exuberant,  5-excessive 

Level  FOUR:  Exercise  of  Safety  Pre-cautions  during  interventions 

Was  the  manipulation  executed  without  risk  of  denuded  tissue? 

(1 -clumsy,  2-marginal,  3 -satisfactory,  4-conservative,  5 -ineffective) 

With  this  classification  there  are  a  possible  (9  manipulations  x  4  levels  x  3  queries  =)  108  variables 
available  for  assessment. 

The  conclusions  are  that  tissue  manipulations  are  the  fundamental  ‘building  blocks’  for  all  of 
surgery.  Generic  surgical  procedures  can  be  constructed  from  standard  choreography  of  these 
manipulations;  however  patient-specific  procedures  require  custom  choreography  of  manipulations. 
Technical  judgment  is  required  for  competence  in  performing  manipulations  and  their  choreography  and 
these  are  subject  to  objective  assessment.  The  manipulations  can  be  classified  into  4  levels  of 
assessment  -  approach,  execution,  consequence,  and  exercise  of  safety  precautions. 

A  hystero-resectoscoe  being  developed  by  Heinrichs  and  colleagues  at  the  Stanford/NASA 
Biocomputation  Center  and  Immersion  Medical  incorporates  these  fundamental  manipulations. 

LTS2000  Laparoscopic  Training  Simulator  and  Objective  Scoring  System  (Hasson) 

Learning  the  visual-motor  skills  that  are  essential  to  performing  video-assisted  surgery  is  best 
achieved  through  practicing  on  an  accentuated  physical  simulator.  The  haptics  or  force-feedback 
technology  of  today’s  virtual  simulators,  while  promising,  is  not  currently  robust  enough  to  provide  the 
same  degree  of  training.  The  LTS2000  (laparoscopy  training  simulator)  (figure  72)  was  developed  to 
fulfill  the  need  for  both  hand-eye-video  coordination  and  procedural  skills  in  a  cost-effective  fashion. 


Figure  72.  The  LTS2000  Laparoscopic  Training  Simulator,  external  and  internal  views 
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Eleven  exercises  were  designed  to  train  skills  such  as  delicate  manipulations,  circular  motions, 
ductal  cannulation,  suturing  and  knot  tying.  Other  exercises  were  designed  to  train  procedure-specific 
skills  such  as  cyst-removal,  morcellation,  and  dissection.  The  LTS2000  exercises  were  designed  to  train 
three  types  of  laparoscopic  skills:  Coordination,  suturing  &  knot-tying  and  procedure-specific  skills. 
The  tasks  objectives  for  the  coordination  skills  are: 

1 .  Spatial  perception  and  orientation. 

2.  Hand-eye  coordination  using  the  dominant  and  non-dominant  hands  separately. 

3 .  Hand-eye  coordination  using  the  dominant  and  non-dominant  hands  together. 

4.  Appreciation  of  fine  tactile  sensations. 

5.  Ability  to  work  under  mirror  image  conditions  or  reversed  hand-eye  axis. 

These  objectives  are  tested  by  a  pegboard  and  posts  model  (with  bead  placement  on  the  pegboard  and 
rings  on  posts),  3D-2D  translation  (by  passage  of  a  probe  through  a  virtual  cube  to  translate  depth  cues 
to  color  codes)  and  duct  cannulation  (by  passage  of  a  pipe  cleaner  through  a  duct  and  retrieval  from  the 
opposite  side). 

The  task  objectives  for  the  suturing  and  knot-tying  skills  are: 

1 .  Placement  of  interrupted  and  continuous  suture. 

2.  Suture  ligation  of  ductal  structures  using  sliding  loops. 

3.  Performing  sliding  loop  and  square  extra-corporeal  knots. 

4.  Executing  instrument  based  squared  intracorporeal  knots. 

5.  Securing  the  first  and  last  stitches  of  a  continuous  suture  with  stable  knots  * 

6.  Building  and  tying  preformed  knots  such  as  the  rolling  hitch  knot. 

These  objectives  are  tested  by  an  external  post  model  (forming  a  Roeder  loop  extra-corporeal  knot), 
silastic  tube  (practicing  loop  knot  tying),  blood  vessel  model  (selective  placement  of  hemostatic  suture 
and  tying  of  knots),  and  sponge  cloth  model  (placing  of  various  stitches  and  practicing  different  knots). 
The  task  objectives  for  procedure-specific  skills  are: 

1 .  Application  of  single  staples  and  clips. 

2.  Practice  with  a  mechanical  linear  stapler. 

3.  Performing  sharp  and  blunt  dissection. 

4.  Practice  fine  dissection. 

5.  Practice  tissue  shaping  and  morcellation. 

6.  Practice  placement  of  a  cyst  into  a  bag  for  extraction. 
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An  objective  scoring  system  was  implemented  to  assess  the  skill  level  of  individual  trainees  before 
and  after  training.  Eleven  exercises  were  selected  and  each  was  assigned  an  equal  point  value  of  100. 
Six  were  scored  in  number  of  tasks  in  1  minute,  and  5  in  number  of  minutes  per  1  task.  A  score  of  100 
was  given  for  completing  a  target  number  of  tasks  in  1  minute  or  the  assigned  task  in  a  target  amount  of 
time.  A  score  of  0  was  given  for  not  completing  a  single  task  in  1  minute  of  for  completing  the  assigned 
task  in  3  times  the  target  time.  Scores  between  100  and  0  were  set  linearly  as  a  function  of  those  values. 
This  system  allows  for  the  scoring  of  results  for  each  trainee  in  each  exercise  as  well  as  providing  an 
overall  score.  Regression  analysis  is  used  to  estimate  the  impact  (pre-test  versus  post-test)  of  each  hour 
of  training  with  the  LTS2000,  controlling  for  the  years  of  experience  of  the  trainees. 

Two  studies  have  been  conducted  to  measure  degree  of  improvement  of  laparoscopic  skills  after 
training  with  the  LTS2000  using  objective  pre-  and  post-training  assessments.  The  studies  involved  39 
physician  users  of  the  training  system;  23  at  a  University  Hospital  I  Kiel  and  16  with  a  University 
affiliated  community  hospital  in  Chicago  (figure  73).  The  physicians  had  various  degrees  of  experience. 
Fifteen  were  attending  surgeons  and  24  were  residents  in  training.  Prior  to  training,  34  of  the  39 
physicians  scored  below  the  75th  percentile  of  the  maximum  achievable  score.  After  training,  only  5 
scored  below  that  level,  which  may  be  considered  a  benchmark. 


Average 

Range 

Pre-Test  Point 

534.5 

45.0-920.8 

Score 

(SD  216.7) 

Post-Test  Point  Score 

957.1 

(SD  126.6) 

549-1100 

Hours  of  Practice 

8.7 

2-30 

Figure  73.  Results  of  the  training  session  with  LTS  2000 


It  can  be  concluded  that  the  scoring  system  used  with  the  LTS2000  is  objective  and  allows 
unbiased  impact  analysis.  Preliminary  results  suggest  that  skill  components  associated  with  advanced 
laparoscopic  surgery  may  be  significantly  improved  through  instructions  and  sustained  practice  on  an 
accentuated  physical  simulator  such  as  the  LTS2000.  The  system  is  being  extended  to  include  a 
mechatronic  component  that  allows  data  reflecting  selected  skills  to  be  displayed  for  immediate 
feedback  to  the  users,  and  collected  for  comparison  and  analysis. 
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What  Makes  a  Skilled  Practioner:  Learning  to  be  a  Doctor  (Krummel) 

The  American  Board  of  Medical  Specialties  (ABMS)  along  with  the  Accreditation  Council  on 
Graduate  Medical  Education  (ACGME)  have  come  to  a  consensus  on  the  6  components  of  surgical 
competence  (figure  74).  Note  that  technical  skills  are  within  Patient  care  and  Practice-based  learning. 
They  are: 


•  Medical  Knowledge 

•  Patient  Care 

•  Interpersonal  &  Communication  Skills 

•  Professionalism 

•  Practice-based  Leaming/Improvement 

•  System-based  Practice 

Figure  74.  Components  of  surgical  competence 

At  the  Center  for  Advanced  Technology  in  Surgery  at  Stanford  (CATSS)  there  are  4  different 
simulators:  Human  Patient  Simulator™  ,  bronchoscopy  trainer,  anastomosis  trainer  and  CathSim™. 

The  simulators  have  been  evaluated  using  a  standard  assessment  protocol  for  usability,  validity  (content, 
construct  and  criterion),  transferability,  comparison  to  existing  modalities  and  cost-benefit  analysis. 

The  anasotomosis  simulator  (figure  75)  is  a  virtual  reality  (VR)  simulator  which  has  a  high  fidelity 
3-D  graphic  computer  generated  representation  of  a  hollow  structure  (intestine,  blood  vessel,  ureter,  etc) 
and  surgical  instruments,  and  a  haptic  interface  using  two  Phantom  haptic  device  (figure  76).  The 
simulation  has  physics-based  tissue  interactions,  and  analysis  software  to  measure  and  evaluate 
performance  of  an  open  surgery  anastomosis.  In  order  to  determine  if  such  a  simulator  can  teach  and/or 
measure  surgical  skills,  a  study  with  8  experienced  surgeons  and  12  medical  students  was  conducted. 
The  8  parameters  measured  were:  Tissue  damage,  accuracy,  peak  tearing  force,  time  to  completion, 
surface  damage,  angular  error,  tool  tip  travel  distance  and  overall  error.  The  study  also  accounted  for 
dominant  versus  non-dominant  hand,  3-D  needle  guidance  and  control  of  needle  holder.  The 
performance  analysis  software  automatically  tabulated  the  participant’s  score  and  printed  out  a  report  at 
the  completion  of  the  procedure  (figure  77).  There  is  also  an  option  to  have  a  real-time  graphic  analysis 
of  the  amount  of  pressure  which  is  being  used  in  order  to  provide  instant  feedback  (“proximate 
learning”). 
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Figure  75  Anastomosis  Simulator  Figure  76  Phantom  haptic  interface 


The  results  of  the  study  demonstrated  construct  validity  since  the  experienced  surgeons 
performed  better  than  students,  and  training  transfer  in  that  6  of  7  students  demonstrated  improvement 
with  repeated  trials.  Some  limitations  of  the  study  are  small  sample  size  which  did  not  completely 
validate  the  proposed  metrics  as  measures  of  surgical  skills,  whether  there  is  transfer  of  skills  to  the  OR, 
and  the  device  was  a  prototype.  There  is  also  the  question  of  the  part-task  trainer.  “It  is  not  true  that  a 
complex  reaction  consists  of  a  chain  of  separate  processes  which  may  be  arbitrarily  added  and 
subtracted.”  L.S.  Vygotsky,  “Mind  in  Society”). 


Figure  77  Automatic  report  generated  at  end  of  trial.  Figure  78  Consequence  of  poor  training 
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Even  seemingly  simple  tasks  like  starting  an  intravenous  (IV)  catheter  can  have  serious  errors 
and  consequences.  In  December,  1997  a  physician  at  a  prominent  medical  training  institution  lost 
litigation  for  infecting  a  patient  with  AIDS  after  a  needle  stick  .  The  .  jury  cites  faulty  medical 
training. . .”  as  the  cause.  A  commercially  available  simulator  from  Immersion  Medical  Corporation  to 
train  the  starting  of  an  intravenous  (IV)  -  CathSim™  has  been  studied  by  the  SurgSim  group  at  Stanford 
University  Medical  Center.  A  total  of  1 14  students  (novice  =28,  medical  student  =  42,  and  registered 
nurses  (RN)  =  44)  participated  in  the  evaluation  of  CathSim™  In  the  novice  study  (n=28),  the  group 
all  had  a  didactic  session,  then  divided  into  two  groups  -  CathSim™  and  mannequin  simulator  -  then 
started  an  IV  catheter  on  a  real  a  person..  The  CathSim™  trained  students  were  better  at  advancing  the 
catheter  on  real  patients.  The  mannequin  group  reduced  subjective  rating  from  pre  to  post  real 
procedure  for  feeling  of  preparedness  to  do  a  real  procedure  (p=.043)  and  usefulness  of  mannequin  for 
training  (p=.002).  The  conclusions  were  that  skill  transfers  from  CathSim  to  real  procedure  were  better 
than  from  the  mannequin  arm  to  real  procedure  in  at  least  one  skill  and  the  CathSim™  promotes  a  more 
realistic  understanding  of  the  procedure  and  the  student’s  ability  level.  This  study  also  established 
usability  and  transferability. 

In  the  medical  student  group  (beginner  =14,  intermediate  =10  and  expert  =17),  the  experts 
performed  better  than  intermediates,  who  performed  better  than  beginners  (construct  validity).  The  test 
results  were:  Same  needle  re-stick  (p=.019),  extra  needle-catheter  units  used  (p=.002),  tourniquet  time 
(p=.000),  pain  factor  (p=.022)  and  misuse  of  tools  (p=.002).  In  addition  content  validity  was  established 
-  expert  physicians  felt  simulator  experience  resembled  the  way  an  IV  should  be  performed  and  was 
realistic  and  useful  (graphic  images,  mock  catheter,  haptic  feedback  of  skin  traction  and  needle 
insertion). 

In  the  nurse  study  (beginners  =16,  intermediates  =9  and  experts  =  19),  the  experts  performed 
better  than  intermediates,  who  performed  better  than  beginners  (construct  validity).  In  addition,  needle 
recannulations  (p=0.036)  and  misuse  of  tools  (p=  .003)  were  in  the  same  trend  from  expert  to  beginner. 

It  must  be  emphasized  that  the  technical  skills  are  but  one  component  which  complements  the 
entire  spectrum  that  comprises  surgical  competence.  Not  only  are  there  psychomotor,  visio-spatial  and 
perceptual  skills,  but  also  knowledge,  judgment,  analysis,  thought  about  action  and  translation  of 
thought  into  action.  It  is  critical  to  build  on  what  medical  education  does  right  and  use  the  power  of 
simulation  to  make  it  better. 
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Fundamentals  of  Laparoscopic  Surgery  (SAGES).  A  New  Standard  in  Education 
for  Laparoscopic  Surgery  (Fried) 

In  reviewing  the  rationale  for  training  in  laparoscopic  surgery,  it  is  found  that  beyond 
laparoscopic  cholecystectomy,  residents’  experience  in  laparoscopy  is  extremely  variable  with  a  few 
programs  providing  an  excellent  experience  in  advanced  procedures,  but  the  majority  having  no 
advanced  experience  or  unequal  exposure  on  one  or  two  services.  Approximately  10%  of  the  graduating 
resident’s  case  experience  is  laparoscopic;  however,  only  1%  of  their  experience  is  in  laparoscopic 
procedures  other  than  laparoscopic  cholecystectomy  and  individual  experiences  in  advanced  procedures 
vary  widely.  A  survey  by  the  Society  of  American  Gastrointestinal  Endoscopic  Surgeons  (SAGES) 
reveals  that  despite  the  best  educational  efforts  of  academic  groups  like  SAGES  there  are  still  only  a 
few  laparoscopic  “champions”  as  educators,  poor  patient  and  referring  physician  awareness  of  less 
invasive  surgical  options,  a  paucity  of  outcomes  data  documenting  effectiveness  of  laparoscopic  surgery 
and  a  lack  of  comfort  with  laparoscopy  as  a  surgical  tool.  Surgeons  reported  that  they  are  uncomfortable 
in  pursuing  advanced  laparoscopic  procedures  because  they  learned  laparoscopy  by  memorizing  the 
techniques  for  a  few  procedures  and  because  there  may  be  a  low  level  of  understanding  and  comfort 
with  the  basics  of  laparoscopy. 

The  rationale  behind  the  SAGES  Fundamentals  of  Laparoscopic  Surgery  (FLS)  course  is  if 
residents  and  their  teachers  can  be  taught  the  underlying  fundamentals  of  laparoscopy,  including 
cognitive  and  motor  skills,  they  will  be  more  comfortable  with  the  use  of  these  techniques  as  a  general 
surgical  tool.  This  will  lead  to  wider  use  and  improved  patient  care.  The  model  is  the  ATLS  course 
because  it  has  a  basic  instructional  course  in  trauma  care  that  teaches  standardized  basic  information 
with  hands-on  training  to  develop  motor  skills  and  had  validated  pre-  and  post-testing  to  document 
learning.  Therefore  the  FLS  was  designed  as  an  integrated  study  course  that  includes  an  instructional 
(CD  ROM  based)  guide  that  verbally  and  visually  teaches  the  fundamental  components  of  laparoscopic 
surgery.  It  includes  situational  interactive  case  studies  to  develop  judgment  skills  and  has  a  pre-  and 
post-op  testing  module  to  grade  cognitive  learning.  In  addition,  FLS  incorporates  a  manual  skills  testing 
module  to  allow  practice  and  assessment  of  basic  laparoscopic  maneuvers  designed  to  increase  the  eye- 
hand  skills  needed  to  work  in  the  video  endoscopic  surgical  milieu.  These  tasks  are  non-procedure 
specific. 

The  Didactic  Section  contains  a  CD  ROM  with  study  guide,  test  and  reference.  There  are 
didactic  chapters  (well  illustrated),  pre-  and  post-tests  (via  Internet),  interactive  case  scenarios  and 
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manual  skills  test  introduction.  The  Manual  Skill  Station  is  a  simple  laparoscopy  box  tied  to  CD  ROM 
for  “watch  &  do”  exercises  with  measurable  exercises  to  quantify  ability  and  improve  skills. 


Figure  79  Identification  of  equipment  and  laparoscopic  instruments. 

The  components  of  the  Didactic  Section  are:  Identification  of  instruments/equipment  (figure  79), 
recognition  of  complications  (figure  80),  anesthesia  position  for  laparoscopic  surgery  (figure  81),  and 
recognition  of  laparoscopic  findings.  There  are  case  scenarios  which  are  designed  to  familiarize  and  test 
for  access  methods  in  laparoscopy,  equipment  needs,  interpretation  of  findings,  management  of 
complications,  trouble  shooting,  physiology  and  pre-  and  post-  operative  care. 


Figure  80  Complications  Figure  81  Anesthesia  positioning 

In  testing  and  validation  of  the  didactic  session  there  is  a  stringent  process  which  resulted  in  the 
creation  of  questions  which  were  analyzed  for  content  validity  (determined  by  review  by  panel  of 
experts)  and  construct  validity  (Does  test  measure  competence  by  showing  differences  in  test  scores 
between  competent  and  non-competent  surgeons?).  The  content  was  created  by  a  comprehensive  review 
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in  defined  subject  areas.  Each  question  is  tested  for  relevance  and  is  agreed  upon  by  consensus  defined 
by  reference  group  of  experts. 

Validation  is  currently  being  conducted  in  a  multi-institutional  study  of  beta  sites  with  a 
reference  group  of  experts.  The  purpose  is  to  compare  scores  between  competent  and  non-competent 
laparoscopic  surgeons,  however  it  may  be  necessary  first  to  define  competent  and  non-competent  groups 
by  some  other  criteria  and  attempt  to  relate  score  on  individual  questions  to  total  score. 

The  goals  for  the  FLS  are  to  provide  a  uniform  curriculum  for  the  acquisition  of  basic  knowledge 
and  technical  skills  in  laparoscopy.  The  FLS  course  will  be  a  reliable  and  valid  process  for  certification 
of  competence  in  basic  laparoscopy  which  may  become  a  new  model  for  training  as  more  technology- 
dependent  surgical  advances  are  introduced. 

Surgical  Competence:  A  Perspective  of  American  Board  of  Medical  Specialties 
(Narhwold) 

There  is  a  growing  public  dissatisfaction  with  the  medical  profession  subsequent  to  publicizing 
of  medical  errors  as  well  as  the  variations  in  care  and  perception  of  poor  service.  The  public  is 
demanding  “I  want  a  competent  doctor”.  In  response  to  the  need  for  accountability,  the  American  Board 
of  Medical  Specialties  (ABMS)  investigated  what  constitutes  medical  competency. 

In  order  to  accurately  identify  the  requirements  of  competency,  it  is  necessary  to  look  at  the  four 
segments  of  a  physician’s  life.  Initially  knowledge  and  degree/certification  begins  with  premedical 
school  resulting  in  an  undergraduate  baccalaureate  degree.  Medical  school  leads  to  a  medical  doctor 
(MD)  degree.  Residency  training  results  in  state  licensure.  Once  in  practice,  certification  is  required. 

The  responsibility  of  degree  granting,  accreditation  and  certification  is  conducted  by  different  bodies 
during  the  4  stages  of  development.  In  premedical  school,  the  university  grants  the  degree,  whereas 
after  medical  school  there  is  the  Liaisons  Committee  on  Medical  Education  (LCME).  During  residency, 
the  Accreditation  Council  on  Graduate  Medical  Education  (ACGME)  and  Residency  Review  Committee 
(RRC)  are  the  responsible  authorities.  During  practice,  the  individual  professional  societies  of  the 
ABMS  are  the  responsible  authorities. 

To  date,  certification  comes  with  an  examination  which  is  independent  of  the  physician’  s 
practice.  There  must  be  an  overhaul  of  the  system  with  a  linkage  between  education  and  accreditation 
and  certification.  There  must  be  a  connection  between  the  process  of  evaluation  of  competency  and 
outcomes 
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The  ABMS  and  ACGME  have  agreed  upon  a  description  of  general  competency  (figure  82). 
The  critical  features  are:  1.  Knowledge,  2.  Patient  care,  3.  Interpersonal  and  communication  skill,  4. 
Professionalism,  5.  Practice-based  learning  &  improvement,  6.  System-based  practice. 


General  Competencies 

1 .  Knowledge 

Maintenance  of  Certification 

2.  Patient  care 

1.  Evidence  of  professional  standing. 

3.  Interpersonal  and  communication  skills 

2.  Evidence  of  lifelong  learning  and 

4.  Professionalism 

quality  improvement. 

5.  Practice-based  learning  &  improvement 

3.  Evidence  of  cognitive  expertise. 

6.  System-based  practice 

4.  Evidence  of  practice  performance. 

Figure  82  General  Competencies  Figure  83.  Maintenance  of  certification 


The  American  Board  of  Medical  Specialties  is  an  umbrella  organization  representing  24  member 
boards  which  issue  112  certificates.  A  consensus  report  included  a  description  of  the  competent 
physician  as  follows: 

“The  competent  physician  possesses  the  medical 
knowledge,  judgment,  professionalism,  and  clinical  and 
communication  skills  to  provide  high  quality  patient 
care.  ” 


Patient  care  encompasses  the  promotion  of  health,  prevention  of  disease,  and  diagnosis, 
treatment  and  management  of  medical  conditions  with  compassion  and  respect  for  patient  and  their 
families.  Certification  of  patient  care  skills  should  include  periodic  oral  examination,  record  review, 
morbidity  and  mortality  conference,  direct  observation,  case/procedure  log,  hospital  credentialing  and 
SIMULATION.  Simulation  provides  a  method  by  which  measurements  can  quantify  performance.  A 
point  to  remember  is  that  “What  we  measure  we  tend  to  improve.”  (David  Leach,  MD  -  Director, 
ACGME).  In  addition,  the  maintenance  of  competence  should  be  demonstrated  throughout  the 
physician’s  career  by  evidence  of  lifelong  learning  and  ongoing  improvement  of  practice  (figure  83). 


Surgical  Competence:  The  Medium  is  the  Message.  A  Perspective  of  the 
Accreditation  Council  of  Graduate  Medical  Education  (Dunn) 

As  a  background  on  the  Accreditation  Council  of  Graduate  Medical  Education  (ACGME),  the 
responsibility  and  mission  is  to  improve  the  quality  of  health  care  in  the  United  States  by  ensuring  and 
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improving  the  quality  of  graduate  medical  educational  experiences  for  physicians  in  training.  The 
ACGME  accredits  7,800  residency  programs  in  26  core  specialties  and  77  subspecialties,  utilizing  peer 
review  process  by  26  Residency  Review  Committees  (RRC),  including  General  Surgery  as  one  of  the 
core  specialties. 

Until  recently,  the  process  (or  “old  system”)  was  basically  an  audit  methodology,  evaluating 
comprised  such  aspects  as  “Does  the  program  comply  with  the  Requirements?”,  “Does  the  program 
have  established  goals  and  objectives  and  an  organized  curriculum?”,  and  “Does  the  program  evaluate 
its  residents  and  itself?”  So  even  though  a  program  met  the  above  criteria,  modem  educational 
processes  have  recognized  the  importance  of  objective  measurements  and  outcomes  analysis.  To  that 
end,  the  ACGME  has  instituted  a  new  Outcomes  Project.  This  is  a  long-term  initiative  to  enhance 
residency  education  through  educational  outcome  assessment.  Instead  of  the  audit  approach  above,  the 
project  addresses  issues  such  as  “Do  the  residents  achieve  the  learning  objectives  set  by  the  program?, 
“What  evidence  can  the  program  provide  that  they  do  so?”,  and  “How  does  the  program  demonstrate 
continuous  improvement  in  its  educational  processes?”  This  requires  a  new  strategy  to  identify  and 
quantify  what  to  measure,  to  develop  measurement  tools  and  to  collaborate  to  find  the  answers. 

In  order  to  identify  what  to  measure,  an  extensive  review  of  literature  (2500  articles)  with  an 
initial  list  of  84  competencies  was  conducted.  An  Advisory  Committee  gave  counsel,  and  there  was 
extensive  vetting  (RRC  members,  program  directors,  residents,  corporate  leaders,  university  presidents, 
public).  Support  was  acquired  through  the  Robert  Wood  Johnson  Foundation.  The  areas  to  be  measured 
include:  Patient  Care,  medical  knowledge,  practice-based  learning  and  improvement,  interpersonal  and 
communication  skills,  professionalism  and  systems-based  practice.  Each  assessment  method  included 
the  measures  of  description,  use,  psychometric  qualities,  feasibility/practicality  and  references.  This 
review  resulted  in  a  report  and  web-based  educational  instrument  entitled  “Toolbox  of  Assessment 
Methods”  which  was  issued  jointly  by  the  ACGME  and  the  American  Board  of  Medical  Specialties 
(ABMS).  For  example,  the  toolkit  on  the  use  of  simulators  indicates  (figure  84): 

In  order  to  measure  technical  competency,  there  must  be  a  model  of  mental  activities  in  directed  skill 
acquisition  at  specific  levels:  Novice,  competence,  proficiency,  expertise  and  mastery  (Stuart  and 
Herbert  Dreyfus).  At  the  novice  level,  the  beginner  is  given  “rules”  for  basis  of  action  in  a  context-free 
experience  and  progresses  by  bringing  behavior  into  conformity  with  the  “rules”.  At  the  competence 
level  and  with  experience  over  time,  the  student  observes  patterns  and  learns  “guidelines”  instead  of 
discrete  rules.  Simulation  is  especially  useful  at  these  two  beginning  levels.  By  the  proficiency  level, 
the  surgeon  should  be  prepared  to  deal  with  a  wide  variety  of  situations  and  understand  the 
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SIMULATIONS  AND  MODELS  Page  13 

DESCRIPTION 

Simulations  used  for  assessment  of  clinical  performance  closely  resemble  reality  and 
attempt  to  imitate  but  not  duplicate  real  clinical  problems.  Key  attributes  of  simulations  are 
that:  they  incorporate  a  wide  array  of  options  resembling  reality,  allow  examinees  to  reason 
through  a  clinical  problem  with  little  or  no  cueing,  permit  examinees  to  make  life-threatening 
errors  without  hurting  a  real  patient,  provide  instant  feedback  so  examinees  can  correct  a 
mistaken  action,  and  rate  examinees’  performance  on  clinical  problems  that  are  difficult  or 
impossible  to  evaluate  effectively  in  other  circumstances.  Simulation  formats  have  been 
developed  as  paper-and-pencil  branching  problems  (patient  management  problems  or  PMPs), 
computerized  versions  of  PMPs  called  clinical  case  simulations  (CCX®),  role-playing 
situations  (e.g.,  standardized  patients  (SPs),  clinical  team  simulations),... 

Figure  84  Example  of  simulation  webpage  of  ACGME  from  Toolkit  of  Assessment  Methods 

procedure  in  terms  of  relevance  for  long-term  goal.  At  this  time  “Principles  and  Maxims”  are  used. 
Expertise  implies  a  repertoire  of  experience  so  vast  that  situations  dictate  an  intuitively  appropriate 
action.  Ultimately  there  is  the  achievement  of  mastery  to  a  point  that  learned  skills  can  cope  with  novel 
and  unpredictable  responses  from  complex  adaptive  systems.  In  a  complex  adaptive  systems  the  parts 
have  the  freedom  and  ability  to  respond  to  stimuli  in  many  different  and  fundamentally  unpredictable 
ways.  A  distinction  must  be  made  between  systems  that  are  largely  mechanical  and  those  that  are 
naturally  adaptive;  the  simulated  surgical  patient  is  an  example  of  the  former;  the  living  patient  an 
example  of  the  latter. 

What  is  left  out  of  our  assessments  may  be  as  important  as  what  is  included.  Each  technology,  in 
itself  valuable  for  teaching  and  assessment,  should  be  kept  in  proper  perspective  and  the  relationship  to 
the  larger  integrated  whole  —  the  result  is  the  competent  physician. 

In  attempting  to  define  the  competent  physician,  the  ACGME  and  ABMS  joint  outcomes  project 
decided  that  it  is  nearly  impossible  to  establish  a  concrete  definition;  therefore  a  description  is  that  the 
competent  physician  should  possess  the  medical  knowledge,  judgment,  professionalism,  and  clinical  and 
communication  skills  to  provide  high-quality  patient  care.  Patient  care  encompasses  the  promotion  of 
health,  prevention  of  disease,  and  diagnosis,  treatment,  and  management  of  medical  conditions  with 
compassion  and  respect  for  patients  and  their  families.  Maintenance  of  competence  should  be 
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demonstrated  throughout  the  physician’s  career  of  lifelong  learning  and  on-going  improvement  of 
practice. 

Goal  II:  DEFINITIONS 

The  definitions  of  what  are  the  specific  meaning  and  connotations  within  the  context  of 
objective  measurements  is  best  derived  from  the  science  of  behavioral  psychology,  which  has  had 
decades  of  research  in  this  area.  Many  subtopics  have  been  spawned,  such  as  psychometrics, 
ergonomics,  man-machine  interface,  human  interface  technologies,  etc,  and  each  has  slightly 
differing  nuances.  Thus,  the  definitions  for  basic  words  such  as  ability,  etc  were  taken  from  the  New 
Webster’s  Dictionary  of  the  English  Language  (Deluxe  Encyclopedic  Edition)  Delaire  Publishing, 

Inc  NYC;  1981  (Appendix  2),  while  those  definitions  for  validity  and  reliability  were  taken  from  : 
Reber,  AS:  Dictionary  of  Psychology.  (Appendix  3). 

There  was  much  debate  about  the  definition  of  “ability”,  which  in  the  behavioral  psychology 
and  objective  assessment  literature  specifically  refers  to  those  characteristics  with  which  a  person  is 
bom  or  develops  in  childhood  and  which,  in  general,  are  not  subject  to  improvement  by  training 
exercises.  Unfortunately  there  are  numerous  other  connotations  for  abilities,  specifically  implying 
the  converse  of  “disability”,  with  the  many  legal  ramifications  of  excluding  a  person  because  they  do 
not  possess  a  high  enough  quality  of  fundamental  abilities  and  therefore  could  seek  refuge  and 
compensation  under  the  Americans  With  Disabilities  Act,  to  mention  just  a  few  potential  conflicts. 
Nevertheless,  it  was  emphasized  that  everyone  has  abilities,  it  is  just  that  some  people  have  more  than 
others.  The  term  abilities  is  so  entrenched  in  the  scientific  literature  that,  though  being  mindful  of  the 
potential  non-scientific  implications  of  the  term,  it  was  a  non-consensus  (though  majority)  opinion 
that  the  term  abilities  most  appropriately  defines  what  was  scientifically  intended  (aptitude  was 
suggested  as  an  alternative,  though  the  connotation  here  was  more  of  a  potential  than  a  measurable 
level  of  physical  capabilities). 

“Skills”  are  the  fundamental  abilities  which  are  brought  together  through  training  to  create 
the  basic  psychomotor  actions  which  a  person  possesses  and  which  are  improved  with  training. 
“Tasks”  are  a  set  of  skills  which  are  used  to  perform  a  specific  action  (such  as  an  anastomosis), 
whereas  a  “procedure”  is  all  the  tasks  choreographed  together  to  create  a  surgical  operation. 

A  common  method  of  scientifically  analyzing  specific  (surgical)  procedures  is  called  “task 
deconstruction”.  The  presumption  is  that  complete  procedures  (like  cholecystectomy)  are  comprised 
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of  a  number  of  “tasks”  (like  isolating  the  cystic  duct)  which  in  turn  is  composed  of  a  series  of  “skills” 
(like  bimanual  dexterity,  transfer,  traversal,  blunt  dissection,  clamping)  and  these  are  based  upon 
fundamental  psychomotor,  visio-spatial  and  perceptual  abilities  (such  as  precision  aiming,  tracking, 
hand-arm  steadiness,  eye-hand  coordination  speed,  etc).  Thus  the  overall  structure  of  increasing 
complexity  mentioned  above  is  abilities,  to  skills,  to  tasks,  to  procedures.  This  progression  will 
provide  a  set  of  definitions  which  can  be  applied  the  various  simulators  to  determine  the  specific 
exercise  (training  or  testing)  level  of  complexity  and  provide  insight  as  to  where  in  the  training  of 
technical  skills  that  the  exercise  can  be  applied. 

In  recognizing  that  technical  skills  are  just  a  portion  of  the  overall  composition  of  “technical 
competency”  and  that  technical  competency  was  just  a  portion  of  a  surgical  competency.  A  number 
of  definitions  related  to  competency  that  were  referred  to  during  the  presentation  of  the  simulation 
systems  were  also  defined,  in  accordance  with  the  same  dictionary  terminology  as  above.  These 
terms  included  competent/competence,  proficient/proficiency,  exercise  and  system  (Appendix  2). 

The  workshop  therefore  has  specific  definitions  which  were  used  in  the  context  of  technical  skills  but 
based  upon  a  broader  definition  to  real  world  usage. 

Goal  III:  TAXONOMY 

A  taxonomy  (Greek  taxis  =  arrangement  and  nomie  =  method)  is  a  method  of  arranging  or 
classifying  larger  entities,  groups  or  systems  into  an  ordered  system,  usually  in  a  hierarchical  fashion 
that  indicates  natural  relationships.  Once  the  definitions  of  the  individual  words  to  be  used  were  agreed 
upon,  other  components  of  simulation  were  brought  into  the  overall  classification.  Although 
recognizing  the  importance  of  overall  competence,  the  workshop  focused  only  upon  developing  a 
taxonomy  for  technical  skill,  and  specifically  the  manual  (but  not  the  cognitive  nor  behavioral/social 
skills).  Thus  the  derived  taxonomy  (Appendix  4)  includes  abilities,  skills,  tasks  and  procedures  as 
applied  to  their  use  by  educational  systems  for  training  and  objective  assessment  of  surgical  skills. 

More  specifically,  the  derived  taxonomy  only  contains  those  exercises  which  currently  are  employed  by 
simulators  available  today.  It  is  acknowledged  that  this  list  is  not  comprehensive,  and  that  some  future 
simulators  will  add  new  abilities,  skills,  tasks  or  procedures.  It  is  envisioned  that,  with  this  baseline 
taxonomy,  future  researchers  will  refer  to  the  categories  and  designate  where  the  components  of  their 
simulators  fit.  The  ultimate  goal  (development  of  the  core  curriculum)  is  that  the  exercises  from  the 
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simulators  will  fit  into  a  “toolbox”  which  educators  and  surgical  program  directors  can  use  in 
establishing  a  training  and  evaluation  curriculum. 

Goal  IV:  MATCHING  METRICS  TO  CURRENT  SYSTEMS 

A  total  of  13  different  simulation  systems  (from  synthetic  materials  in  open  surgery  drills,  to 
animal  parts  in  open  and  laparoscopic  systems,  to  full  virtual  reality  in  both  open  and  laparoscopic 
skills)  that  were  reviewed  in  the  first  session  were  matched  against  the  measurements  (metrics)  that  were 
identified  in  the  taxonomy  (Appendix  5).  This  synthesis  provides  a  “look-up  table”  for  program 
directors  and  other  educators  in  helping  create  a  training  curriculum  that  is  comprehensive  (i.e.  which 
covers  the  basic  abilities,  skills,  tasks  and  procedures)  within  the  context  of  what  is  available  today. 

Since  no  single  system  covers  all  the  components  of  surgical  technical  skills,  a  number  of  different 
methods  and  simulators  will  need  to  be  combined  to  provide  the  learning  experience  for  the 
resident/registrar.  In  reviewing  what  is  available,  it  is  apparent  that  only  a  few  exercises  are  available 
for  fundamental  abilities  (though  they  are  very  well  validated)  while  there  is  a  plethora  of  simulations 
for  skills  and  tasks.  Full  procedures  are  too  complex  to  simulate,  other  than  in  performing  them  in  an 
animal  model.  Simple,  low  fidelity  systems  (from  box  trainers  to  virtual  reality)  can  cover  a  significant 
part  of  the  early  learning  for  a  full  procedure  (egg,  cholecystectomy),  however  it  will  be  a  significant 
time  frame  before  that  level  of  sophistication  can  be  brought  to  the  personal  computer  level. 

Goal  V:  DEVELOPMENT  OF  A  CORE  CURRICULUM 

There  was  lively  debate  about  deciding  upon  a  core  curriculum.  One  of  the  major  difficulties  is  that 
different  training  programs  (especially  on  an  international  basis)  have  different  rates  at  which  students 
are  introduced  to  technical  skills.  There  are  important  differences  which  make  the  creation  of  a 
“standard”  core  curriculum  very  difficult  at  this  time  in  the  developing  science  of  simulation.  Therefore, 
it  was  agreed  that  classification  of  training/evaluation  should  be  according  to  three  levels  of  difficulty  - 
basic,  intermediate  and  advanced  (Appendix  6).  Abilities,  skills,  tasks  and  procedures  were  therefore 
categorized  as  to  whether  they  were  basic,  intermediate  or  advanced  level  of  difficulty  and  clinical 
applicability.  By  this  approach  offers  a  toolbox  of  exercises  (similar  to  the  ACGME  toolkit)  with  which 
training  program  directors  can  customize  their  training/evaluation  program  according  to  their 
educational  needs  and  even  (perhaps)  to  the  skill  (and  learning  curve)  of  the  individual  student  or  group 
of  students  (egg  PGY  2,  3).  Over  time,  as  more  simulators  become  available  and  validated,  it  may  be 
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possible  to  create  a  standard  core  curriculum  for  an  (international  training  program.  However  the 
nascent  field  of  objective  assessment  has  not  progressed  far  enough  at  this  time  to  have  the  outcomes 
that  would  scientifically  support  one  specific  curriculum.  It  may  also  be  possible  that  this  toolbox  can 
be  incorporated  into  larger  efforts,  such  as  the  ACGME,  ABMS  or  NBME  in  the  United  States  to 
provide  the  technical  component  to  compliment  the  overall  assessment  of  competence. 

CONCLUSION 

The  essential  goals  of  the  workshop  to  establish  a  consensus  on  definitions  and  taxonomy 
(classification)  of  specific  educational,  training  and  evaluation  exercises  that  are  available  in  today’s 
simulators  were  met.  The  goal  of  establishing  a  curriculum  was  not  met;  instead,  levels  of  training 
(basic,  intermediate  and  advanced)  with  simulation  exercises  available  at  each  level,  were  provided  in 
order  that  individual  training  program  directors  could  construct  and  customize  a  training  program 
dependant  upon  local  needs. 

There  are  a  few  identified  areas  for  future  research.  While  realizing  this  is  not  a 
comprehensive  list,  it  represents  a  few  glaring  absences  in  the  current  arena  that  should  have  near- 
term  research  for  resolution.  While  definitions  for  many  aspects  were  proposed  and  agreed  upon, 
there  was  only  a  speculation  and  listing  (from  “brainstorming”)  on  what  constitutes  an  error 
(Appendix  8),  what  is  a  taxonomy  for  errors,  and  how  errors  should  be  measured.  In  addition,  there 
were  three  distinct  areas  for  research  that  were  identified  during  the  workshop.  1).  A  number  of 
surgeons  identified  a  need  for  a  skill  called  “tissue  handling”.  None  of  the  systems  measure  such  a 
skill.  2).  There  are  only  a  few  exercises  which  evaluate  fundamental  abilities;  it  is  known  that  there 
are  many  such  tests  available  in  the  non-medical  arena  and  therefore  these  should  be  identified  and 
validated.  3).  Comparison  and  integration  of  the  various  available  exercises  of  the  different  systems 
into  a  single  coherent  “core  curriculum”. 

It  is  anticipated  that,  as  the  field  of  objective  assessment  using  simulators  matures  with 
validation  by  outcomes  analysis  at  the  end  of  the  training  programs,  there  will  be  a  convergence 
toward  a  core  curriculum  that  can  provide  a  standardization  for  surgical  resident  training  and 
evaluation  as  well  as  certification  -  all  directed  at  reducing  errors  to  improve  the  quality  of  patient 
safety. 


83 


Metrics  Conference 


1  August  2, 2001 
Version  1.3 


ADMINISTRATIVE  ACTIONS 

The  After- Action  for  this  workshop  is  illustrated  in  the  timelines  and  deliverables  (Appendix 
9),  which  includes  providing  a  final  report  by  1  September,  2001.  Circulation  to  numerous  relevant 
societies  and  agencies  for  vetting  (Appendix  10)  will  occur  prior  to  setting  an  Open  Forum  of  the 
general  surgical  education  community.  A  Board  of  Advisors  was  elected.  (Appendix  1 1)  and  an 
Agenda  of  the  workshop  is  included  for  archival  purposes. 
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he  maintains  a  strong  interest  and  involvement  in  UGI  and  HPB  surgery.  He  has  been  prominent  in 
promoting  and  achieving  new  standards  of  trauma  care  at  State  and  National  levels  in  Australia  for 
more  than  15  years  and  has  lectured  extensively  on  trauma  subjects  in  Australia  and  internationally. 
He  led  the  EMST  (ATLS®)  programme  in  ANZ  from  1987  to  1996.  He  is  an  active  member  of  the 
American  and  International  Trauma  Surgery  Associations.  Responsibilities  and  recognition  within 
the  Royal  Australasian  College  of  Surgeons  (RACS)  have  included  membership  of  Trauma 
Committee,  RACS  Foundation  Lecturer  for  Trauma  Section  of  Annual  Scientific  Congress  1996, 
recipient  of  RACS  Medal  in  1997  and  election  to  Council  of  the  RACS  in  1998.  He  took  up 
Chairmanship  of  the  RACS  Board  of  Basic  Surgical  Training  in  May  2001 .  He  is  the  Australian  and 
New  Zealand  representative  on  the  Board  of  Governors  of  the  American  College  of  Surgeons.  He  is 
President  Elect  of  IATSIC  (International  Association  for  the  Surgery  of  Trauma  and  Surgical 
Intensive  Care).  Professor  Deane  has  chaired  two  RACS  Committees  which  are  involved  in  new 
initiatives  in  surgical  skills  training  and  surgical  skills  laboratory  development.  He  has  had  a  key 
role  in  the  early  development  of  the  Definitive  Surgical  Trauma  Care  (DSTC)  Course  within 
IATSIC  and  the  Australian  surgical  community. 
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DUNN,  Marvin  R.,  M.D 

Dr.  Dunn  is  a  board  certified  pathologist  and  currently  the  Director  of  RRC  Activities  for  the 
ACGME.  Dr.  Dunn  has  previously  served  as  the  Director  of  the  Division  of  Graduate  Medical 
Education  and  Secretary  to  the  Council  on  Medical  Education  of  the  American  Medical  Association, 
and  as  Dean  at  both  the  University  of  South  Florida  College  of  Medicine  and  the  University  of 
Texas  Medical  School  at  San  Antonio. 

FRIED,  Gerald  M.,  M.D.  FACS 

Dr.  Fried  graduated  from  McGill  University  Faculty  of  Medicine  in  1975.  He  then  completed 
his  residency  in  general  surgery  at  McGill  University  in  1980,  which  included  one  year  of  training  in 
gastrointestinal  surgery  at  Ohio  State  University  under  the  supervision  of  Dr.  Larry  Carey  and  Dr. 
Robert  Zollinger.  He  then  completed  a  research  fellowship  in  gastrointestinal  endocrinology  at  The 
University  of  Texas  Medical  Branch,  Galveston,  under  the  direction  of  Dr.  James  C.  Thompson. 

This  involved  the  study  of  cholecystokinin  and  its  the  physiologic  effects  on  the  biliary  tract  and 
pancreas. 

Dr.  Fried  returned  to  McGill  University  and  The  Montreal  General  Hospital  in  July  1982, 
and  has  remained  there  since.  He  has  maintained  research  funding  for  most  of  that  time  for  basic  or 
clinical  research,  and  has  been  involved  actively  in  surgical  education.  He  was  Program  Director  for 
General  Surgery  from  January  1991-December  1996,  and  Fellowship  Director  in  Endoscopic 
Surgery  since  1996.  He  is  currently  Professor  of  Surgery,  Associate  Head  of  Surgery  (Clinical)  for 
the  Montreal  General  Hospital  site  of  the  McGill  University  Health  Centre,  Director  of  the  Section 
of  Minimally  Invasive  Surgery  at  McGill  University,  and  Director  of  Gastrointestinal  Surgery  and 
Minimally  Invasive  Surgery  at  The  Montreal  General  Hospital.  He  is  also  Director  of  The  Florenz- 
Steinberg  Bernstein  and  David  Bernstein  Centre  for  Research  and  Education  in  Minimally  Invasive 
Surgery,  and  Director  of  Tyco  Healthcare/United  States  Surgical  Corporation  Centre  of  Excellence 
Program  at  McGill  University. 

He  has  been  Chairman  of  the  Committee  on  Endoscopic  and  Laparoscopic  Surgery  of  the 
Canadian  Association  of  General  Surgeons,  is  currently  Chair  of  the  Patient  Care  Committee  of  The 
Society  for  Surgery  of  The  Alimentary  Tract,  and  co-chair  of  SAGES  committee  on  The 
Fundamentals  of  Laparoscopic  Surgery  Program.  His  current  academic  interests  are  in  training  and 
evaluation  of  laparoscopic  skills,  and  measurement  of  patient-based  outcomes  after  minimally 
invasive  surgery.  His  clinical  interest  is  in  gastrointestinal  surgery,  and  the  multidisciplinary 
application  of  endoscopic,  imaging,  and  minimally  invasive  technologies  in  the  care  of  patients  with 
digestive  diseases. 


GALLAGHER,  Anthony,  PhD 

Dr.  Gallagher  completed  his  BSc  at  the  University  of  Ulster  in  1988  and  then  his  PhD  in 
Psychology  at  Trinity  College  Dublin  in  1992.  He  is  a  Consultant  at  the  Royal  Victoria  Hospital  in 
Belfast,  Director  of  Research  at  the  Northern  Ireland  Center  for  Endoscopic  Training  and  Research 
(Royal  Group  Hospitals  Trust  &  Queen’s  University  Belfast),  founding  member  and  one  of  the 
Directors  of  the  Queen’s  University  Advanced  Informatics  in  Medicine  (AIM)  Research  Institute. 
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He  is  currently  Fulbright  Distinguished  Scholar  at  the  Department  of  Surgery  at  Yale.  In 
minimal  access  surgery  (MAS)  he  has  three  research  areas,  optimal  skill  acquisition  strategies  in 
MAS,  human  factors  in  MAS  and  objective  assessment.  For  the  last  six  years  he  and  his  colleagues 
have  been  investigating  different  training  strategies  which  will  allow  surgical  residents  to  automate 
quickly  to  the  counterintuitive  eye-hand  co-ordination  required  for  MAS.  In  particular  they  have 
been  investigating  the  use  of  virtual  reality  (VR)  trainers.  Their  published  research  evidence  shows 
that  VR  trainers  are  at  least  as  good,  if  not  better,  in  achieving  basic  psychomotor  performance 
levels  which  are  currently  established  on  traditional  box  trainers.  He  and  his  group  have  also  been 
quietly  working  on  the  development  of  an  objective  assessment  protocol  of  fundamental  abilities 
(i.e.,  visio-spatial,  perceptual  &  psychomotor)  for  MAS.  This  protocol  and  a  number  of  empirical 
replications  were  reported  for  the  first  time  at  the  EAES  in  Maastricht  2001 . 

At  the  Dept,  of  Surgery  at  Yale  University  along  with  Faculty  members  (Richard  Satava 
MD,  Dana  Anderson  MD,  Neal  Seymour  MD  and  Sanziana  Roman  MD)  they  have  been 
investigating  whether  training  on  VR  transfers  to  performance  in  the  operating  room.  They  have 
also  been  investigating  how  strongly  objectively  assessed  fundamental  abilities  predict  performance 
in  the  operating  room.  This  study  will  conclude  in  August  2001 . 

HAMDORF,  Jeffrey  M,  MBBS,  Ph  D,  FRACS 

Dr.  Hamdorf  is  a  Senior  Lecturer  in  the  University  Department  of  Surgery  at  the  Queen 
Elizabeth  II  Medical  Centre.  His  interests  include  Gastrointestinal  Surgery,  Upper  Gastrointestinal 
Malignancy  and  Neoadjuvant  chemoradiotherapy  as  well  as  Surgical  Education,  Curriculum 
Development  and  Surgical  Skills  Development 

His  Career  Progression  began  with  Surgical  Research  Scholar,  University  Department  of 
Surgery,  UWA  (1987),  followed  by  Surgical  Registrar,  Interhospital  Rotation,  Western  Australia 
(1989).  In  1993  he  became  a  Visiting  Academic  at  the  University  of  New  South  Wales,  then 
aSurgical  Fellow,  The  Prince  of  Wales  Hospital,  Randwick,  NSW(1994).  Since  1995  he  has  been 
Consultant  Surgeon,  Sir  Charles  Gairdner  Flospital  and  Senior  Lecturer  in  Surgery,  The  University  of 
Western  Australia.  His  teaching  experience  and  appointments  include  University  Department  of 
Surgery  Teaching  Group  (Chair  2000,  Deputy  1998),  Member,  Association  of  Surgical  Educators 
(1999),  Basic  Surgical  Skills  Committee,  Basic  Surgical  Training,  RACS  (Chair  2000,  Deputy 
1998),  State  Supervisor  (Western  Australia),  Basic  Surgical  Training,  RACS  (2000),  Surgical 
Director,  CTEC,  The  Centre  for  Medical  and  Surgical  Skills,  UWA  and  the  Assessment  Committee, 
The  University  of  Western  Australia 

HASSON,  HarrithM.,  MDFACOG 

Dr.  Hasson  graduated  from  Ein  Shams  University,  Cairo,  Egypt,  1955  and  served  internship  at 
Ein  Shams  University,  Cairo,  Egypt-  1955-1956,  and  St.  John's  Episcopal  Hospital,  Brooklyn,  N.Y. 

-  1958-1959.  His  residency  in  Pathology  at  St.  John's  Episcopal  Hospital,  Brooklyn,  N.Y. 

1959- 1960,  and  a  second  residency  in  OB/Gyn,  Presbyterian  St.  Luke's  Hospital,  Chicago,  IL 

1960- 1961,  and  West  Suburban  Hospital  Oak  Park,  IL,  1961-1963  and  1965-19 66.  He  is  certified 
by  the  American  Board  of  Obstetrics  and  Gynecology,  1969,  and  licensed  to  practice  medicine  in  the 
State  of  Illinois.  His  hospital  Appointments  are  Grant  Hospital  of  Chicago,  Chairman,  Department 
of  Obstetrics/Gynecology,  (1981-1995);  Weiss  Memorial  Hospital,  Director,  Gynecologic 
Endoscopy  Center,  (1995-  )Chairman,  Division  of  Obstetrics/Gynecology,  (1996-)  Co-Director, 
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Minimally  Invasive  Surgery  Center,  (1998-).  His  university  affiliations  are  Northwestern  University, 
Assistant  Professor,  Obstetrics  and  Gynecology,  (1976-1981),  Rush  Medical  College,  Associate 
Professor,  Obstetrics  and  Gynecology,  ( 1 98 1  - 1 995)  and  University  of  Chicago,  Clinical  Professor 
Obstetrics  and  Gynecology,  (1996-) 

Dr.  Hassan  has  been  director  and  officer  of  several  organizations  including  the  American 
Association  of  Gynecologic  Laparoscopists  (AAGL),  Society  for  the  Advancement  of  Contraception 
(SAC), -The  Accreditation  Council  of  Gynecologic  Endoscopy  (ACGE)  and  the  Chicago 
Gynecological  Society.  In  addition  he  has  been  Program  Director  for  more  than  20  continuing 
medical  education  programs  related  to  endoscopy  and  gynecology  held  in  the  USA  and  abroad.,  a 
consultant  to  several  organizations  including  International  Fertility  Research  Program,  Planned 
Parenthood  Federation,  Program  for  Applied  Research  on  Fertility  Regulation  and  the  RAND 
Corporation.  He  is  the  current  Editor  of  The  Journal  of  the  American  Association  of  Gynecologic 
Laparoscopists  and  AdHoc  reviewer  for  numerous  journals. 

Dr.  Hasson  has  received  numerous  professional  honors  for  developing  the  Hasson  Technique  of 
open  laparoscopy  and  he  received  certificates  of  Merits  and  awards  from  the  American  Fertility 
Society,  1977,  the  AAGL,  1993,  the  SLS,  2000.  He  is  a  Biographee  in  Who's  Who  in  Technology 
Today  and  Who's  Who  of  American  Inventors.  He  has  published  53  original  articles  in  peer  reviewed 
journals  and  edited  a  book  on  IUDs  and  wrote  24  chapters  on  several  subjects  including:  open 
laparoscopy,  selective  fitting  of  IUDs,  laparoscopic  ovarian  surgery,  laparoscopic  myomectomy,  and 
laparoscopic  hysterectomy.  He  has  also  served  as  a  visiting  professor  to  many  University  Hospitals  in 
North  America,  South  America,  Europe,  the  Middle  East  and  Australia  and  produced  30  films  and 
video  presentations  on  surgical  procedures  shown  in  the  USA  and  abroad.  He  now  holds  13  grants 
primarily  for  research  in  surgical  instruments  and  procedures.  And  has  received  51  patents  for  surgical 
instruments  with  2  more  pending  applications. 


HEINRICHS,  William  LeRoy,  MD„  PhD. 


Dr.  Heinrichs  is  a  Professor  and  Chair  (Emeritus,  Active)of  Gynecology  &  Obstetrics  of 
Stanford  University  Medical  Center.  His  education  and  training  include  Southwestern  State 
University  in  Oklahomo,  University  of  Oklahoma  School  of  Medicine  and  MS/PhD  at  University  of 
Oregon  Medical  Sciences  University 


PROFESSIONAL  TRAINING: 

1958- 1959  Rotating  Intern,  St.  Anthony  Hospital,  Oklahoma  City,  OK(affiliate  U.  of  Okla.) 

1959- 1961  Assistant  Resident  in  Obstetrics  and  Gynecology,  Harper  Hospital,  Detroit,  MI 

1961-1962  Senior  Resident  in  Obstetrics  and  Gynecology,  Harper  Hospital,  Detroit,  MI 

(affiliate  of  Wayne  State  University  School  of  Medicine) 


RESEARCH  AND/OR  PROFESSIONAL  EXPERIENCE: 

1965-1967  Clinical  Assistant  Professor,  Department  of  Obstetrics  and  Gynecology,  Univ.  of 
Oregon  School  of  Medicine,  Portland,  OR 

1967-1969  Assistant  Professor  of  Obstetrics  and  Gynecology,  Univ.  of  Washington  School  of 
Medicine,  Seattle,  WA 

1 969- 1972  Associate  Professor  of  Obstetrics  and  Gynecology,  Univ.  of  Wash.  Sch.  of  Med. 

1972-1976  Professor  of  Obstetrics  and  Gynecology,  Univ.  of  Wash.  School  of  Medicine 
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1976-1984  Professor  and  Chairman,  Department  of  Gynecology  and  Obstetrics,  Stanford 
University  School  of  Medicine,  Stanford,  CA 

1976-1984  Chief  of  Service,  Department  of  Gynecol  &  Obstet,  Stanford  University  Hospital 
1976-1984  Member,  Executive  Committee,  Stanford  University  School  of  Medicine 
1976-1984  Member,  Clinical  Advisory  Committee,  Stanford  University  Hospital 

1984- 1 994  Professor  of  Obstetrics  and  Gynecology,  Stanford  University  of  Medicine 

1985- 1 994  Director  of  Gynecology,  Department  of  Gynecology  and  Obstetrics 

1994-present  Professor  and  Chairman  (Emeritus,  Recalled  1999),  Department  of  Gynecology 

and  Obstetrics,  Stanford  University  School  of  Medicine 
HONORS  (Selected): 

1958  Medical  Degree  with  Honors:  Student  Research  Achievement  Award  (Biochemistry) 

1966  Appointment  as  Josiah  Macy,  Jr.  Faculty  Fellow 

1970  President’s  Award,  Am.  College  of  Obstet.  and  Gynecol.  (Estrogen  Receptors  in 

Brain) 

1970-1978  Associate,  Acting  Editor,  Gynecologic  Investigation,  an  International  Journal  of  the 
Science  of  Reproduction,  Karger,  Publ.,  Basel 

1974-1978  Member,  PHS  (NIH);  Study  Section,  Human  Embryology  and  Development  (HED) 

1979-1981  Chairman,  PHS  (NIH);  Study  Section,  Human  Embryology  and  Development  (HED) 

1978  Chairman,  Reproductive  Dysfunction  Study  Group,  SCOMSEC  -  Scientific  Group  on 
Methodologies  for  the  Safety  Evaluation  of  Chemicals;  sponsored  by  Scientific 
Committee  on  Problems  of  the  Environment  (SCOPE),  and  WHO,  Lago  Majore, 
ITALY 

1980  Invited  Presentation,  NIH  Consensus  Development  Conference  on  Magnetic 
Resonance  Imaging:  “Reproduction  Toxicity  Studies  of  Gamete  and  Embryotoxicity, 
and  Teratogenesis”,  Bethesda,  MD 

1998  Chairman;  Women’s  Healthcare  Session,  Medicine  Meets  Virtual  Reality VII,  San 
Francisco 

1999  Keynote  Speaker,  IEEE/CBMS,  Stamford,  CT,  June  '99,  “From  Slices  to  Surgical 
Simulation:  What  Do  Surgeons  Want?” 

2000  Keynote  Speaker,  MMVR,  Newport  Beach,  CA,  Jan.'OO,  "Envisioning  Healing" 
Organizer:  Stereoscopic  Image  Session,  at  MMVR,  Newport  Beach,  CA,  Jan.'OO 
Invited  Speaker:  NIH/NICHD  Planning  Group  (Pregnancy  Research)  oh  Nausea  & 
Vomiting  in  Pregnancy,  Bethesda,  MD,  Oct.'OO 

KAUFMANN,  Christoph  R.,  MD,  MPH,  FACS,  COL,  MC,  USA 

COL  Kaufmann  is  a  graduate  of  the  Uniformed  Services  University  of  the  Health  Sciences. 
He  completed  his  general  surgery  residency  at  Tripler  Army  Medical  Center  in  1987  and  a 
trauma/critical  care  fellowship  at  Harborview  Medical  Center,  University  of  Washington  in  1989. 

He  is  board  certified  in  both  general  surgery  and  surgical  critical  care.  He  is  an  Associate  Professor 
of  Surgery  and  Military  &  Emergency  Medicine  at  the  Uniformed  Services  University  of  the  Health 
Sciences. 

He  was  deployed  with  the  47th  Combat  Support  Hospital  to  Desert  Shield/Desert  Storm 
including  Saudi  Arabia  and  Iraq  for  which  he  was  awarded  the  Bronze  Star.  He  served  as  the 
Director,  Division  of  Trauma  and  Emergency  Medical  Systems,  Health  Resources  and  Services 
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Administration,  USPHS,  a  $5  million  annual  federal  grant  program.  In  1995,  he  was  awarded  the 
Diploma  in  the  Medical  Care  of  Catastrophes  (DMCC)  by  the  Society  of  Apothecaries  of  London. 

He  currently  serves  as  Chief,  Division  of  Trauma  and  Combat  Surgery,  USUHS  and  Chair, 
American  College  of  Surgeons  Army  Committee  on  Trauma,  with  responsibility  for  all  Advanced 
Trauma  Life  Support  courses  taught  in  the  Army  and  in  Triservice  institutions.  Dr.  Kaufmann  is  the 
Principle  Investigator  for  the  DoD  Testbed  for  Telepresence  Surgery  at  USUHS,  with  applications 
of  advanced  technology  to  surgical  education  being  a  focus  of  his  research  over  the  past  two  years. 
He  is  the  Director  of  the  Surgical  Simulation  Lab  of  the  National  Capital  Area  Medical  Simulation 
Center.  Dr.  Kaufmann  publishes  and  presents  nationally  and  internationally  on  trauma  and  surgical 
simulation. 

KRUMMEL,  Thomas  M.,  MD,  FACS 

Dr.  Krummel  is  currently  is  the  Emile  Holman  Professor  and  Chair  of  the  Department  of 
Surgery  of  theStanford  University  School  of  Medicine.  He  is  a  native  of  Wisconsin  and  completed 
his  undergraduate  degree  at  the  University  of  Wisconsin  with  a  medical  degree  at  the  Medical 
College  of  Wisconsin.  His  surgical  residency  was  at  the  Medical  College  of  Virginia  with  a 
Fellowship  in  Pediatric  Surgery  at  the  Children’s  Hospital  of  Pittsburgh  followed  by  a  Research 
Fellowship  both  at  MCV  and  UCSF.  Following  five  years  on  the  faculty  at  MCV  he  was  named 
Professor  of  Surgery  and  Chief  of  Pediatric  Surgery  at  the  Pennsylvania  State  University  College  of 
Medicine  and  Surgeon-in-Chief  at  the  Children’s  Hospital  in  1990.  In  1994  he  was  named  John  A. 
and  Marian  T.  Waldhausen  Professor  and  Chair  of  the  Department  of  Surgery  and  Surgeon-in-Chief 
at  University  Hospitals.  In  1998  he  moved  to  Stanford  to  assume  the  position  as  Emile  Holman 
Professor  of  Surgery  and  Chairman  of  the  Department  of  Surgery  at  Stanford  University. 

Dr.  Krummel  is  a  member  of  a  number  of  professional  societies  including  the  American 
College  of  Surgeons,  the  American  Pediatric  Surgical  Association,  the  Society  for  Clinical  Surgery, 
and  the  American  Surgical  Association.  In  1997  he  was  named  a  Director  of  the  American  Board  of 
Surgery.  He  has  served  in  leadership  positions  in  many  surgical  organizations,  and  has  been 
fortunate  to  have  participated  in  the  training  of  a  number  of  medical  students  and  residents  who  have 
pursued  productive  careers  in  Surgery.  Over  the  last  five  years  he  has  been  a  pioneer  in  the 
application  of  information  technology  to  enhance  the  quality  and  safety  of  surgical  education  and 
reduce  its  staggering  cost.  In  collaboration  with  computer  scientists,  engineers  and  industry  he  has 
participated  in  the  development  of  several  surgical  trainers  and  has  begun  the  systematic  study  of 
their  use  and  efficacy  in  surgical  education. 

MAGEE,  J.  Harvey 

Mr.  Magee  has  a  diverse  background  of  30  years’  successful  professional  experience,  spanning 
both  military  service  and  private  sector  business  experience.  A  hospital  administrator  by  profession, 
Mr.  Magee  retired  from  the  United  States  Air  Force  Medical  Service  Corps  in  1996  at  the  rank  of 
Major.  Upon  his  retirement  from  the  USAF  Surgeon  General’s  Office  in  1996,  Mr.  Magee  was  the 
Mid  Atlantic  Regional  Manager  for  PROMODEL,  Healthcare  Services  Division,  representing 
MedModel,  and  then  joined  SHERIKON,  Inc.  in  June  2000  and  serves  as  the  Project  Officer  for 
Surgical  Technology  and  Medical  Modeling  and  Simulation  (MM&S),  at  the  US  Army’s 
Telemedicine  and  Advanced  Technology  Research  Center  (TATRC),  HQ  US  Army  Medical 
Research  and  Materiel  Command  (USAMRMC),  Fort  Detrick,  Maryland.  He  coordinates  a  diverse 
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Research  &  Development  (R&D)  portfolio  of  advanced  surgical  and  MM&S  projects.  He  is 
assigned  as  the  Project  Officer  for  TATRC  Medical  Modeling  and  Simulation  initiatives 

Mr.  Magee  graduated  from  the  University  of  Mississippi  (“Ole  Miss”)  in  1970,  with  a  Bachelors 
of  Business  Administration  (BBA)  in  General  Business  and  completed  Air  Command  and  Staff 
College  in  1988  and  Squadron  Officer  School  in  1987.  Mr.  Magee  was  certified  by  the  Federal 
Aviation  Administration  1992  as  a  Total  Quality  Management  (TQM)  Facilitator.  During  his  two 
separate  tours  of  active  duty,  Maj  Magee  received  the  Meritorious  Service  Medal,  First  Oak  Leaf 
Cluster;  Meritorious  Service  Medal,  Reserve  Medal,  and  Commendation  Medal.  He  was  nominated 
for  Alaskan  Air  Command  Officer  of  the  Year  in  1992  and  nominated  for  Young  Federal  Health 
Care  Administrator  in  1991. 

From  1970  -  1973,  Lt  Magee  was  assigned  to  USAF  Hospital  Tyndall,  Panama  City,  Florida, 
serving  as  Director,  Medical  Materiel  and  Director,  Resource  Management.  From  1973-1976,  Capt 
Magee  was  assigned  to  USAF  Hospital  Bitburg,  West  Germany,  serving  as  Director,  Resource 
Management  and  Squadron  Commander.  And  from  1976  -  1978,  Capt.  Magee  served  at  the 
Command  Surgeon’s  Office,  HQ  US  Air  Forces  in  Europe.  He  was  Chief,  Administrative  Support 
Services,  monitoring  the  administrative  effectiveness  of  24  USAF  medical  facilities  in  Europe.  From 
1978  -  1979,  Mr.  Magee  was  the  Western  Regional  Administrator  for  The  Navigators,  Inc., 
Colorado  Springs,  Colorado.  The  Navigators  is  an  international,  interdenominational  Christian 
service  organization  with  representatives  in  more  than  30  countries  and  from  1979  -  1986,  Mr. 
Magee  developed,  and  then  managed,  a  private  athletic  and  fitness  facility  in  Rapid  City,  South 
Dakota  and  also  served  as  an  Individual  Mobilization  Augmentee  (IMA)  in  the  USAF  Reserves  at 
USAF  Hospital  Ellsworth,  Ellsworth  AFB,  Rapid  City,  South  Dakota.  From  1986  -  1989,  after 
eight  years  off  of  active  duty  in  private  business,  Maj  Magee  accepted  a  voluntary  recall  to  active 
duty  in  1986.  He  was  assigned  to  the  Air  Force’s  2nd  largest  medical  center,  USAF  Medical  Center 
Wright-Patterson,  Dayton,  Ohio,  as  Director  of  Patient  Administration.  .  From  1989  -  1993,  Maj. 
Magee  was  assigned  to  Elmendorf  AFB,  Alaska,  serving  as  Director  of  Resource  Management, 
Director  of  Patient  Administration,  and  Associate  Administrator.  From  1993  -  1996,  Mr.  Magee 
was  assigned  to  the  United  States  Air  Force  Office  of  the  Surgeon  General,  Washington,  DC,  to 
manage  the  Wartime  Medical  Planning  System  (WAR-MED),  Fort  Detrick,  Maryland.  WAR-MED 
is  a  $15  million  Advanced  Research  and  Development  program  simulating  casualty  treatment  and 
evacuation  worldwide.  Mr.  Magee  spearheaded  and  coordinated  development  efforts  of  the 
acquisition  community,  identified  financial  and  manpower  resources  necessary  to  operate  and 
sustain  WAR-MED’s  deployment,  and  prepared  the  medical  community  to  deploy  and  implement 
this  simulation  technology 

NAHRWOLD,  David  L.,  MD  FACS 

Dr.  Nahrwold  received  his  undergraduate  and  medical  degrees  from  Indiana  University, 
where  he  also  completed  residency  training  in  surgery.  After  service  on  the  faculties  at  Indiana  and 
Penn  State,  he  was  the  Loyal  and  Edith  Davis  Professor  and  Chairman  of  the  Department  of  Surgery 
at  Northwestern  University  for  15  years. 

A  gastrointestinal  surgeon,  his  clinical  and  basic  research  interests  are  in  the  gastrointestinal 
field,  about  which  he  has  written  extensively.  His  bibliography  lists  more  than  200  publications.  He 
has  served  on  the  editorial  boards  of  nine  periodicals  and  is  Editor  Emeritus  of  the  Journal  of 
Laparoendoscopic  and  Advanced  Surgical  Techniques. 
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Dr.  Nahrwold  has  held  elected  offices  in  numerous  societies  and  organizations,  including  the 
American  Board  of  Surgery,  of  which  he  was  Chairman.  Currently  Emeritus  Professor  of  Surgery  at 
Northwestern,  he  most  recently  served  as  Interim  Director  of  the  American  College  of  Surgeons.  He 
is  a  regent  of  the  College  and  a  member  of  the  executive  committee  of  the  American  Board  of 
Medical  Specialties,  of  which  he  is  also  president-elect. 

REGHER,  Glenn,  PhD 

Dr.  Regher  received  his  PhD  in  experimental  cognitive  psychology  from  McMaster 
University  in  Hamilton,  Ontario,  Canada  in  1993  under  the  supervision  of  Dr.  Lee  Brooks.  His 
thesis  work  focused  on  the  development  of  expertise  in  visual  classification  tasks  (such  as 
dermatological  diagnosis).  He  trained  for  one  year  as  a  research  associate  in  medical  education  with 
Dr.  Geoff  Norman  at  McMaster  University  Medical  Centre,  and  joined  the  University  of  Toronto 
Faculty  of  Medicine  as  an  assistant  professor  and  education  researcher  in  August,  1993.  In  June 
1998,  he  was  appointed  as  the  first  Associate  Director  of  the  newly  established  University  of 
Toronto  Faculty  of  Medicine  Centre  for  Research  in  Education  at  the  University  Health  Network. 

Currently,  as  well  as  being  the  Associate  Director  of  the  CRE,  Dr.  Regehr  is  Associate 
Professor  in  the  Departments  of  Psychiatry  and  Surgery  in  the  University  of  Toronto  Faculty  of 
Medicine,  Associate  Faculty  in  the  University  of  Toronto  Department  of  Education  and  is  a  Scientist 
in  the  Toronto  General  Research  Institute  at  the  University  Health  Network.  He  has  chaired  the 
Association  of  Canadian  Medical  Colleges  Committee  on  Research  in  Medical  Education  for  the  last 
four  years  and  has  chaired  the  Association  for  Surgical  Education  Research  Committee  for  the  last  2 
years.  He  sits  on  the  editorial  boards  of  Advances  in  Health  Sciences  Education  and  Medical 
Education,  and  is  a  member  of  the  Editorial  Research  Advisory  Committee  for  Academic  Medicine. 

Dr.  Regehr  has  researched  and  published  in  a  variety  of  content  domains  including,  the 
evaluation  of  selection  procedures  for  undergraduate  and  post-graduate  training  programs,  the 
development  of  new  methodologies  for  studying  self-assessment  ability  in  trainees,  the  refinement 
of  clinical  skills  assessment  tools  such  as  the  Objective  Structured  Clinical  Exam,  and  the  teaching 
and  testing  of  technical  skills  in  surgery.  He  has  extensive  theoretical  training  and  practical 
experience  in  experimental  and  evaluation  research  methodologies  and  in  inferential  and 
psychometric  statistical  techniques. 


ROSSER,  James  C.  MD  FACS 


Dr.  James  “Butch”  Rosser,  Jr.  was  bom  in  1954  in  Rome,  Mississippi,  but  despite  growing 
up  in  the  segregated  south,  he  refused  to  be  burdened  by  its  racial,  financial,  and  social  barriers.  He 
received  his  undergraduate  degree  in  chemistry  and  biology  from  the  University  of  Mississippi.  He 
completed  his  medical  training  at  the  University  of  Mississippi  School  of  Medicine  before 
completing  a  five-year  surgical  residency  at  Akron  General  Medical  Center,  where  in  1984-85  he 
served  as  Chief  Resident.  After  his  residency,  Dr.  Rosser  began  a  private  surgical  practice  at  Akron 
General  Medical  Center  and  accepted  a  position  as  Assistant  Professor  of  Surgery  at  Northeastern 
Ohio  Universities  College  of  Medicine,  where  he  received  the  1991  “Golden  Apple  Professor  of  the 
Year”  award  for  his  outstanding  contributions  to  medical  education.  Early  in  his  career,  inspired  by 
Dr.  Herbert  Awender,  Dr.  Rosser  realized  the  potential  of  endoscopic  and  minimally  invasive 
surgery.  He  has  pioneered  a  number  of  minimally  invasive  procedures,  most  notably  his  streamlined 
laparoscopic  suturing  technique,  and  now  travels  the  globe  teaching  his  Laparoscopic  Suturing 
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Symposium  and  other  techniques  to  thousands  of  surgeons.  He  has  also  distinguished  himself  by 
performing  laparoscopic  cholecystectomy  procedures  on  some  of  the  youngest  individuals  in  the 
world  (15, 17,  and  19  months),  which  earned  him  Kent  State  University’s  “Minority  Achievement 
Award.” 

Currently,  Dr.  Rosser  is  Associate  Professor  and  Director  of  Endo-Laparoscopic  Surgery  at 
Yale  University.  He  has  been  a  contributing  editor  of  Surgical  Laparoscopy  and  Endosconv.  a 
moderator  at  the  Fourth  World  Endoscopic  Congress,  and  chairperson  of  the  minimally  invasive 
post-graduate  course  for  the  American  College  of  Surgeons,  Society  of  American  Gastrointestinal 
Endoscopic  Surgeons  (SAGES),  American  Medical  Association  and  Southern  Medical  Association. 
Dr.  Rosser  has  written  two  books  due  to  be  released  this  fall,  several  chapters  in  major  laparoscopic 
textbooks,  and  over  forty  surgical  journal  articles.  His  medical  research  interests  are  mini¬ 
laparoscopy  under  local  anesthesia,  conscious  pain  mapping,  nerve  injury  syndromes,  long-term 
central  venous  catheters,  and  continuous  dynamic  monitoring  during  endoscopic  and  laparoscopic 
procedures.  Dr.  Rosser  has  authored  eight  CD-ROM  titles  over  the  last  five  years,  and  six  of  them 
will  be  released  next  spring  under  the  “Yale  University  Laparoscopic  Series”  label.  His  efforts  have 
not  only  been  directed  toward  surgeons,  but  have  also  broken  new  ground  in  the  area  of  patient 
information  and  informed  consent,  gaining  critical  acclaim  in  both  the  New  England  Journal  of 
Medicine  and  its  European  counterpart  The  Lancet. 

As  the  founder  of  the  non-profit  organization  Modem  Day  Miracle  Incorporated,  Dr. 
Rosser’s  goal  is  to  expose  the  ‘modem  day  miracle’  of  minimally  invasive  surgery  to 
underprivileged  and  undereducated  countries  around  the  world,  many  times  via  telemedicine,  the 
remote  care  of  patients  using  modem  telecommunications.  His  “Operation  Outreach”  pioneered  the 
technique  of  remotely  guiding  surgeons  with  little  or  no  experience,  who  ultimately  performed 
successful  advanced  laparoscopic  procedures  with  Dr.  Rosser  thousands  of  miles  away,  and  he  feels 
that  the  safe  maturation  of  telemedicine  should  be  an  important  aspect  of  21st  century  healthcare. 
Modem  Day  Miracle  Incorporated  provides  follow-up  inspection,  data  gathering,  and  continuing 
education  tours  to  countries  in  need,  as  well  as  here  in  the  United  States,  thereby  allowing  the  art 
form  of  laparoscopic  surgery  to  be  properly  nurtured.  The  program  has  been  implemented  in 
Greece,  Jamaica,  Aruba,  and  other  Caribbean  countries.  Dr.  Rosser’s  research  and  development 
activities  have  been  featured  on  the  Learning  Channel,  the  CBS  Morning  Show,  CNN,  and  the 
Discovery  Channel,  and  he  has  garnered  three  Smithsonian  Awards. 


SATAVA,  Richard  M,  MD  MS  (Surg  Rsch),  FACS 

Dr.  Satava,  MD,  is  a  practicing  general  surgeon  specializing  in  advanced  medical  technology 
research.  He  is  also  Professor  of  Surgery  at  Yale  University  School  of  Medicine.  Dr.  Satava  is 
retired  from  the  active  duty  Army  Medical  Corps  and  was  previously  the  program  manager  at  the 
Defense  Advanced  Research  Projects  Agency  (DARPA).  In  addition  to  serving  on  the  American 
College  of  Surgeons’  Regents  Committee  on  Informatics,  Dr.  Satava  serves  as  a  member  of  the 
College’s  Emerging  Technologies  and  Residents  Education  Committees.  He  received  his 
undergraduate  degree  at  John  Hopkins  University,  Baltimore,  and  he  received  his  medical  degree 
from  Hahnemann  University,  Philadelphia.  Dr.  Satava  served  and  internship  at  the  Cleveland 
Clinic,  his  surgical  residency  at  the  Mayo  Clinic,  Rochester,  MN  and  he  completed  a  fellowship 
with  a  Masters  of  Surgical  Research  at  the  Mayo  Clinic.  He  has  been  a  member  of  the  White  House 
Office  of  Science  and  Technology  Policy  (OSTP),  Committee  on  Health,  Food  and  Safety.  In 
)  addition,  Dr.  Satava  is  the  past-president  of  the  Society  of  American  Gastrointestinal  Endoscopic 
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Surgeons  (SAGES),  he  is  a  member  of  the  board  of  governors  of  the  Society  of  Minimally  Invasive 
Therapy,  The  Society  of  Laparoendoscopic  Surgeons  (SLS)  and  he  is  active  in  numerous  surgical 
and  engineering  societies.  Dr.  Satava  serves  on  the  editorial  board  of  numerous  surgical  and 
scientific  journals  and  he  has  written  extensively  about  surgical  education  and  surgical  research. 


SINANAN,  MikaN.,  MD  FACS 

Dr.  Sinanan  is  currently  an  Associate  Professor  of  Surgery  and  Adjunct  Professor  in  Electrical 
Engineering  at  the  University  of  Washington  School  of  Medicine.  He  received  his  MD  at  the  Johns 
Hopkins  University  School  of  Medicine  and  PhD  in  the  Department  of  Physiology,  University  of 
British  Columbia,  Canada .  He  is  the  Co-Director  of  the  Center  for  Videoendoscopic  Surgery  and 
Chief  if  Medical  Staff  at  the  University  of  Washington  Medical  Center.  He  is  a  dedicated  researcher 
and  frequent  contributor  to  peer  review  publications  in  the  area  of  minimally  invasive  surgery  and 
surgical  education. 


The  “Metrics  Group”  10  July,  2001 


First  Row:  Brian  Clauser,  Carleen  Ellis,  Sir  Alfred  Cuschieri,  David  Nahrwold,  Ara  Darzi,  Patricia  Youngblood,  Jeffrey  Hamdorf, 
Mika  Sinanan,  George  Hanna,  Gehard  Buess.  Second  Row:  Glenn  Regehr,  Ralph  dayman,  Rick  Satava,  Paul  Wetter,  Randy  Haluck, 
Leroy  Heinrichs,  Stephen  Deane,  Anthony  Gallagher,  Jack  Jakimowicz,  Vivek  Datta.  Third  Row:  Dan  Smith,  Butch  Rosser,  Adrian 
Park,  Anthony  DiSciullo,  George  Lewis.  Fourth  Row:  Harvey  Magee,  Gerald  Fried,  Harry  Hasson,  Chris  Kaufmann,  Lee  Sillin. 
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